I’ve been “lucky” to be involved in an Oracle Exadata Cloud@Customer (Generation 2) implementation for a major US retailer who are consolidating around 6 Exadata machines of various generations onto two ExaC@C (one non-production half rack, one production half rack) and it’s been a journey.
Space… It’s been a challenge. It’s not disk space, of course with 350TB of ASM storage and 1.5TB of memory approx per node (4) – it’s the other stuff.
– by default, we had 60,000 semaphores defined (/etc/sysctl.conf – kernel.sem)
– each new database consumes 1500 approx semaphores although processes defaults to 1024
– increased the semaphores from 60k to 180k and noticed about 130k in use but continued to encounter issues
ORA-27300: OS system dependent operation:semget failed with status: 28 ORA-27301: OS failure message: No space left on device ORA-27302: failure occurred at: sskgpcreates Yeah, those...
The tyranny of /var…. Out of the box, Oracle gives it a paltry 5GB and sure, there’s probably a good way of expanding it – but with as the number of databases created grows, this area fills up pretty rapidly. Then you may create a database and find you’re out of space… Whilst I mention it, the /u01 area may be undersized due to a bug.
Disappearing binaries – remember how Oracle are not supposed to be poking around in your environment? Well, mysteriously our 11.2 Oracle homes on /u02 (at least one of which houses active databases) suddenly lost datapatch/sqlpatch. For those who never use datapatch (please write to me with an explanation about how you’ve avoided it because I could learn a lot from you), it’s basically a wrapper for $ORACLE_HOME/sqlpatch/sqlpatch. And that’s missing in action too. The sqlpatch directory is full of patches but no way to run them (other than perhaps cloning an Exadata 184.108.40.206 home and bringing the binaries across).
Disappearing /u01 stuff. Well, where to begin. A while back, the /u01/app/oracle/product directory had a golden image of each Oracle version and downloaded patch set (so 12.2 OCT2020, 12.2 JAN2021, etc). Now, the directory structure is there (11.2.0, 12.1.0, 12.2.0, 18.0.0 and 19.0.0) but they are all empty. Even if you perform a dbaascli command to download specific versions, it remains empty.
Why am I getting issues with logons?
Auto-update is evil