{"id":245,"date":"2013-10-28T11:04:46","date_gmt":"2013-10-28T11:04:46","guid":{"rendered":"http:\/\/tomas.papp.me.uk\/?p=245"},"modified":"2013-10-28T11:04:46","modified_gmt":"2013-10-28T11:04:46","slug":"exadata-related","status":"publish","type":"post","link":"https:\/\/tomas.papp.me.uk\/?p=245","title":{"rendered":"Exadata related…."},"content":{"rendered":"

Background Processes in the Exadata Cell Environment on database server:<\/strong>
\nThe background processes for the database and Oracle ASM instance for an Exadata
\nCell environment are the same as other environments, except for the following background process:<\/p>\n

– diskmon Process – The diskmon process is a fundamental component of Exadata Cell, and is responsible for implementing I\/O fencing.<\/p>\n

– XDMG Process (Exadata Automation Manager)
\nIts primary task is to watch for inaccessible disks and cells, and to detect when the disks and cells become accessible.<\/p>\n

– XDWK Process (Exadata Automation Worker)
\nThe XDWK process begins when asynchronous actions, such as ONLINE, DROP or ADD for an Oracle ASM disk are requested by the XDMG process.
\nThe XDWK process will stop after 5 minutes of inactivity.<\/p>\n

Output:<\/strong>
\n> ps -ef | egrep “diskmon|xdmg|xdwk”
\noracle 4684 4206 0 06:42 pts\/1 00:00:00 egrep diskmon|xdmg|xdwk
\noracle 10321 1 0 2010 ? 00:38:15 \/u01\/app\/11.2.0\/grid\/bin\/diskmon.bin -d -f
\noracle 10858 1 0 2010 ? 00:00:18 asm_xdmg_+ASM1<\/p>\n

As a departure from ASM storage technology that uses a process architecture borrowed from database instances,
\nthe storage servers have a brand new set of processes to manage disk I\/O. They are:<\/strong>
\n– RS, the restart service. Performing a similar role to SMON, RS monitors other processes, and automatically restarts them if they fail unexpectedly.
\nRS also handles planned restarts in conjunction with software updates.
\nThe main cellrssrm process spawns several helper processes, including cellrsbmt, cellrsbkm, cellrsomt, and cellrsssmt.<\/p>\n

– MS, the management service. MS is the back-end process that processes configuration and monitoring commands. It communicates with cellcli, described in the next section.
\nMS is written in Java, unlike the other background processes which are distributed in binary form and are likely written in C.<\/p>\n

– CELLSRV, the cell service. CELLSRV handles the actual I\/O processing of the storage server.
\nIt is not uncommon to see heavy usage from CELLSRV process threads during periods of heavy load.
\nAmong other things, CELLSRV provides:
\n. Communication with database nodes using the iDB\/RDS protocols over the InfiniBand network
\n. Disk I\/O with the underlying cell disks
\n. Offload of SQL processing from database nodes
\n. I\/O resource management, prioritizing I\/O requests based on a defined policy<\/p>\n

– I\/O Resource Manager (IORM). Enables storage grid by prioritizing I\/Os to ensure predictable performance<\/p>\n

Cell node Management Overview:<\/strong>
\nDBA’s login as OS user “celladmin” to manage cell nodes.
\nEach cell node internally run ASM instance to manage cell node disks. This means, you can’t see the ASM pmon process on the cell node.<\/p>\n

Cell Admin. Tool’s: cellcli and dcli.
\nCell monitoring Tool’s: OSWatcher, ORION (I\/O performance benchmarking tool) and ADRCI<\/p>\n

Cell Nodes Logs and Traces:<\/strong>
\n$ADR_BASE\/diag\/asm\/cell\/`hostname`\/trace\/alert.log
\n$ADR_BASE\/diag\/asm\/cell\/`hostname`\/trace\/ms-odl.*
\n$ADR_BASE\/diag\/asm\/cell\/`hostname`\/trace\/svtrc__0.trc — ps -ef | grep “cellsrv 100”
\n$ADR_BASE\/diag\/asm\/cell\/`hostname`\/incident\/*<\/p>\n

\/var\/log\/messages*, dmesg
\n\/var\/log\/sa\/*
\n\/var\/log\/cellos\/*<\/p>\n

cellcli -e list alerthistory<\/p>\n

$OSSCONF\/cellinit.ora — #CELL Initialization Parameters
\n$OSSCONF\/cell_disk_config.xml
\n$OSSCONF\/griddisk.owners.dat
\n$OSSCONF\/cell_bootstrap.ora<\/p>\n

\/opt\/oracle\/cell\/cellsrv\/deploy\/log\/cellcli.lst*<\/p>\n

$OSSCONF\/alerts.xml
\n$OSSCONF\/metrics\/*
\noswatcher data<\/p>\n

df -h -> check if \/opt\/oracle file system full? \/opt\/oracle only 2GB in size on cell node !!!<\/p>\n

Where :
\n$OSSCONF is: \/opt\/oracle\/cell11.2.1.3.1_LINUX.X64_100818.1\/cellsrv\/deploy\/config
\n$ADR_BASE is: \/opt\/oracle\/cell11.2.1.3.1_LINUX.X64_100818.1\/log<\/p>\n

Cell Check and shutdown\/startup commands:<\/strong>
\nNote: For full list of commands use: cellcli -e help<\/p>\n

cellcli -e alter cell shutdown services all
\ncellcli -e alter cell startup services all
\ncellcli -e alter cell shutdown services cellsrv
\ncellcli -e alter cell restart services cellsrv
\ncellcli -e list lun detail
\ncellcli -e list griddisk detail
\ncellcli -e list celldisk detail
\ncellcli -e list physicaldisk detail
\ncellcli -e list flashcache detail
\ncellcli -e list physicaldisk attributes name, diskType, luns, status
\ncellcli -e list physicaldisk where disktype=harddisk attributes physicalfirmware
\ncellcli -e list lun attributes name, diskType, isSystemLun, status<\/p>\n

imagehistory (root\/sudo)
\nimageinfo (root\/sudo)
\nservice celld status (root\/sudo)
\nlsscsi | grep MARVELL<\/p>\n

Smart scan layers:<\/strong>
\nSmart scan involves multiple layers of code
\nKDS\/KTR\/KCBL – data layers in rdbms
\nKCFIS – smart scan layer in rdbms
\nPredicate Disk – smart scan layer in cellsrv
\nStorage index – IO avoidance optimization in cellsrv
\nFlash IO – IO layer in cellsrv to fetch data from flash cache
\nBlock IO – IO layer in cellsrv to fetch data from hard-disks
\nFPLIB – filtering library in cellsrv<\/p>\n

Is smart scan issue?<\/strong>
\nCell_offload_processing=false (default true)
\nIf the problem does not occurs, it\u2019s the smart scan issue.<\/p>\n

Is this a FPLIB issue?<\/strong>
\n_kcfis_cell_passthru_enabled=true (default false)
\nIf the problem does not occurs, it\u2019s the FPLIB issue<\/p>\n

Is storage index issue?<\/strong>
\n_kcfis_storageidx_disabled=true (default false)
\nProblem still occurs, it\u2019s not a storage index issue.<\/p>\n

Is flash cache issue?<\/strong>
\nFor 11.2.0.2, _kcfis_keep_in_cellfc_enabled=false (default true) do not use flash cache
\nFor 11.2.0.1, _kcfis_control1=1 (default 0)
\nProblem still occurs, it\u2019s not a flash cache problem.<\/p>\n

Cell related Database view’s:<\/strong>
\nselect * from sys.GV_$CELL_STATE;
\nselect * from sys.GV_$CELL;
\nselect * from sys.GV_$CELL_THREAD_HISTORY;
\nselect * from sys.GV_$CELL_REQUEST_TOTALS;
\nselect * from sys.GV_$CELL_CONFIG;<\/p>\n

Bloom filter in Exadata:<\/strong>
\nIn Oracle 10g concept of bloom filtering was introduced.
\nWhen two tables are joined via a hash join, the first table (typically the smaller table) is scanned and the rows that satisfy the \u2018where\u2019 clause predicates (for that table) are used to create a hash table.
\nDuring the hash table creation a bit vector or bloom filter is also created based on the join column.
\nThe bit vector is then sent as an additional predicate to the second table scan.
\nAfter the \u2018where\u2019 clause predicates have been applied to the second table scan, the resulting rows will have their join column hashed and it will be compared to values in the bit vector.
\nIf a match is found in the bit vector that row will be sent to the hash join. If no match is found then the row will be disregarded.
\nOn Exadata the bloom filter or bit vector is passed as an additional predicate so it will be overloaded to the storage cells making bloom filtering very efficient.<\/p>\n

How to Identify a Bloom Filter in an Execution plan:<\/strong>
\nYou can identify a bloom filter in a plan when you see :BF0000 in the Name column of the execution plan.<\/p>\n

To disable the feature, the initialization parameter _bloom_pruning_enabled must be set to FALSE.<\/p>\n","protected":false},"excerpt":{"rendered":"

Background Processes in the Exadata Cell Environment on database server: The background processes for the database and Oracle ASM instance for an Exadata Cell environment are the same as other environments, except for the following background process: – diskmon Process – The diskmon process is a fundamental component of Exadata Cell, and is responsible for […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-245","post","type-post","status-publish","format-standard","hentry","category-oracle"],"_links":{"self":[{"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/posts\/245","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=245"}],"version-history":[{"count":1,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/posts\/245\/revisions"}],"predecessor-version":[{"id":246,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=\/wp\/v2\/posts\/245\/revisions\/246"}],"wp:attachment":[{"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=245"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=245"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tomas.papp.me.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=245"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}