Tuesday, September 10, 2013

ORA-00600 when using EXPDP & IMPDP on a RAC Database.

When exporting metadata (without data) using EXPDP, things often proceed smoothly. However, when performing an IMPDP using the sqlfile mode, some Oracle RAC environments may trigger an ORA-00600 internal error. This is recognized by Oracle Support as a bug (no patch may yet exist), so caution is needed before running IMPDP in production.

The Scenario & Symptoms

  • EXPDP command (metadata-only export) completes without issues.

  • IMPDP with sqlfile mode fails with an internal error:

    ORA-00600: internal error code, arguments: [15851], [4], [4], [1], [3],
  • The alert logs show that Oracle issues “ALTER SYSTEM SET SERVICE_NAMES” during the import attempt, introducing a generated SYS$SYS service name, e.g., SYS$SYS.KUPC$C_....

  • The jobs DM00/ DW00 (Data Pump processes) may attempt operations that lead to the error trace.

  • No lasting impact to the database is typically observed, but the import fails and the sqlfile is not created.

Sample Log Extract (from alert log)

ALTER SYSTEM SET service_names='SRV_JOB','SRV_ORCL','ORCL','SYS$SYS.KUPC$C_4_20130826105315.ORCL' SCOPE=MEMORY SID='ORCL_4'; ALTER SYSTEM SET service_names='SYS$SYS.KUPC$C_4_20130826105315.ORCL','SRV_JOB','SRV_ORCL','ORCL',... SCOPE=MEMORY SID='ORCL_4'; DM00 started ... DW00 started ... ORA-00600: internal error code, arguments: [15851], [4], [4], [1], [3], …

It’s noted that this may be triggered specifically when running IMPDP in a RAC context, due to internal handling of services and metadata injection.

Possible Root Causes (RCA) & Mechanisms

  1. Dynamic Service Registration / Modification
    Data Pump (especially in RAC) may issue or adjust SERVICE_NAMES entries to manage worker processes across instances. The generation of internal service names (SYS$SYS.*) might cause inconsistencies or unexpected internal state transitions in the process.

  2. Bug in Data Pump / RAC Interaction
    The presence of [15851] in the ORA-00600 arguments suggests a known issue in the Oracle kernel or Data Pump internals, where a corner-case in metadata import on RAC nodes triggers an undesired internal assertion or state.

  3. Incomplete Statistics / Missing Fixed Object Stats
    As you observed, running DBMS_STATS.GATHER_FIXED_OBJECTS_STATS() is sometimes proposed as a workaround, but that alone may not fully mask the underlying bug or prevent the failure in all environments.

Workaround & Mitigation Strategies

  • Execute:

    EXEC DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

    This collects statistics on fixed dictionary objects, which might prevent certain internal errors. However, in your tests, this did not reliably mitigate the issue.

  • Use metadata-only EXPDP / IMPDP without sqlfile mode when feasible.

  • Instead of sqlfile, use DIRECT import of metadata (i.e. let IMPDP apply changes rather than generating script), while monitoring carefully in lower environments first.

  • Validate that all RAC instances are stable, with consistent service configurations, before initiating the import.

  • If possible, run the import on a single instance (non-RAC) environment, generate the SQL file there, and then apply it in the RAC environment manually after validation.

  • Contact Oracle Support, referencing the internal error arguments [15851] and your database version/RAC configuration, to see if they have a patch or workaround for your specific version.

Best Practices & Considerations

  • Always test Data Pump operations in lower environments before running in production, especially on RAC architectures.

  • Maintain up-to-date Oracle patches and PSU bundles, because some internal bugs may be fixed in patch sets.

  • Collect full diagnostics: run ADRCTL or ADRCI to collect the incident dump, trace files, and metadata, and provide them to Oracle Support:

    Use ADRCI or Support Workbench to package the incident (see Note 411.1 on MOS).
  • Monitor your trace directory size limits—errors like ORA-48913 (writing to trace file failed due to size limit) suggest your trace file quotas may have been exceeded.

  • Archive trace files and incidents promptly to avoid file system overflow.

  • Validate service name consistency across RAC nodes, and avoid unnecessary dynamic service additions unless absolutely needed.

Summary

  • The ORA-00600 [15851] error during IMPDP … SQLFILE in a RAC environment is a known internal issue triggered by service modifications in Data Pump.

  • While GATHER_FIXED_OBJECTS_STATS may sometimes alleviate the issue, it is not a guaranteed fix.

  • Use alternative import strategies (direct metadata import, non-RAC staging, careful scripting) when possible, and involve Oracle Support with full diagnostics if needed.

No comments: