EXP378 System Manager’s Report

Michael Hodge and Steven Thomas

Servers (Microsoft):

• Encountered issue where Commvault monthly tape report did not list any tapes for extraction. Further investigating indicated tape report was executed prematurely before physical tapes are actually created.  Because of new Spectralogic Black Pearl NAS, installation last expedition and Commvault scheduler utility, monthly backups are run on first Sunday of the new month and then tapes are created later in week after tape report execution.  Solution is to change Commvault’s scheduler for when monthly tape report runs.  However, process is to be changed next expedition so no action taken.

• Opened Commvault SR# 200117-251 about why report generator not finding monthly full backup tapes in tape library. The reason identified were tapes were not marked as completed upon backup completion and were being left in an open state for next monthly backup continual usage.  The solutions was to set parameter to complete tapes after backup (option checked - Mark Media Full on Success).  This resolved matter.

• Another issue discovered is index backups are being recorded to tapes with a misnomer labeling. However, process is to be changed next expedition so no action taken.

• Opened Commvault SR# 200131-473 for scheduled weekly Scratch Tape Report that was listing scratch tapes in and out of tape library, which is not desired. Preference is to see only scratch tapes remaining in tape library.  Requested assistance from Commvault on modifying that report and now reflects scratch tapes in library correctly.

Exchange:

• No issues to report about Exchange email system or servers.

Servers (Linux):

• During Expedition, 368P appears many admin & administrator passwords were changed. On Expedition 383 discovered http://iprint/ipp did not work or display webpage correctly.  SR# 101240400681 was opened with Micro Focus on Expedition 383 to troubleshoot and resolve.  Finally, on this Expedition 378 the problem was identified to how the password for the admin account was changed.  MCS must use a defined script to change the password on the appliance and not use standard tools for password changes.  Utilized the Micro Focus provided password change script tool resolved the matter and now http://iprint/ipp resolves webpage and shows printers that can be installed.

• Encountered out-of-disk space error messages for server Olympus. Discovered Apache-Tomcat directory was consuming all available disk space.  Further investigating revealed on EOX Expedition 385 portcall Apache was upgraded to version 9 and the crontab job that keeps the Apache-Tomcat directory cleared was not revised to reflect the correct directory with new Apache-Tomcat version.  Edited crontab job to reflect correct directory resolved matter.

Servers (ODA):

• Existing ODA servers performed as expected without any issues.

• New Oracle X7-2M ODA servers were shipped to Expedition 378 for installation.

o   Reviewed and edited DNS tables in preparation for ODAs installation.

o   Extracted retired Krakatoa (Rigwatch) server from server rack.

o   Installed new ODAs K1 & K2 into Krakatoa’s vacated rack space and plumbed in all necessary cabling from switch to ODAs along with power.  This included new cable management system shipped with ODAs.

o   Implemented network VLAN for ODAs.

o   Upon power-up encountered ODA flapping issue where ODA continuously reboots.  Installed patch to prevent flapping and concluded ODA physical installation.

o   Built new VMware VM for new Oracle Enterprise Manager (OEM) console for ODAs named OEMJR.

o   Worked with shore personnel and concluded Oracle configuration for LIMJR database for importation into new ODAs.

Servers (VMware):

• VMware servers performed as expected without any issues.

3PAR SAN:

• HP 3PAR SAN performed as expected without any issues.

• Responded to shore request for HP 3PAR SAN specifications and statistics.

o   Current total drive (raw) size.  -- 52TB, approximately

o   Current total yield of available storage.  --19TB free, approximately

o   Current total allocated space. -- 33TB, approximately

Network:

• While investigating a high network bandwidth workstation discovered Unit 3 of the Focsle B5 switch was misconfigured. The switch was set to default role of NONE and Authentication turned off on all port.  Reconfigured and set ports to default role of no internet and Authentication turned on except for printer port and Viji Telemedicine ports.

• For Oracle, database monitoring and synchronizing between ship and shore, setup shore server OEM in JR Sonicwall for connectivity with JR Oracle servers.

• TAMU shore has decommissioned FQDN techbox-old.ads.tamu.edu (old RDP/Exchange Management) server. New management server is techbox.ads.tamu.edu.  Configured JR Sonicwall to open port for new IP address of management server.

• Updated both SMA200s firmware to v10.2.0.0 (maintenance release) and matched configuration between units.

PC Workstations:

• Technician encountered communication issues with AVS PC working. It appears that it was a power-up sequence issue.  Suggest power-up sequence as follows;

o   The instrument needs to be totally powered down,

o   All cables need to be connected,

o   Power up instrument first, power up PC second,

o   Then press the communications button.

• Both Petrel Logging workstations lost their drive mappings to Yellowstone server from recent Windows 10 migration. The SMB 1.0 / CIFS Client had to be re-enabled in Widows 10 Add Features.  Once enabled set the specific folder on Yellowstone server to map to drive letter Z: on local PC and be persistent resolved matter.

• Installed HighScore Plus & PDF-4 Axiom 2019 XRD software on PC53309 in Corelab forward/starboard area. Was able to register HighScore, but not PDF-4 software.  Discovered that ICDD PDF-4 Axiom 2019 software is licensed one workstation per license.  We have two licenses and both have been deployed.  Shore suggested using the older XRD EVA software at this time.

• TAMU shore issued announcement for all Windows OS users to acquire and apply Jan 2020 patch (CVE-2020-0601 | Windows CryptoAPI Spoofing Vulnerability). Approved the patch for download in our WSUS server and awaiting for it to conclude download for disperse.

• Received an Extended Windows 7 support product key for PC52529 SEM instrument and applied key successfully.

• Installed multiple Python and R programs on LO’s secondary PC53268 for training purposes.

• Placed Publications PC53285 into Internet Access group per email directive from shore management. Both PC and MAC are permanent additions to the internet group.

• Meetings between Engineering and Core Technician groups about Drill Shack replacement workstations. Started identifying components for a proposal.

• The recently developed Thin Section Report Writer (TSRW) program does not work correctly at this time. The fix for now is to go back to the old TSRW that was a Java Network Launcher Program (jnlp) file and install Java Runtime Environment (JRE) for those workstations that require TSRW.

• Multiple issues with the PXRF Olympus Handheld XRF X-ray gun laptop LT52531. The laptop was tested successfully last expedition but would not communicate with gun during preliminary testing at BOX of this expedition.  Discovered OS had become deactivated and would not accept OS license key.  The laptop then had to be rebuilt from scratch.  Laptop was rebuilt and confirmed operational by XRD lab technician.

MAC Workstations:

• Due to EOX data size increases over last couple of years, received 1TB SSD drives for Userroom and CoChief’s MACs upgrades. However, upon installation discovered missing SATA cables, drive sled and screws for new enlarged SSD drives.  Found kits with required parts and requisition for next expedition/tie-up period.

• During Expedition 368X encountered scientists with new MAC OS-X Mojave OS that was not compatible with our anti-virus program McAfee End Point Security (EPS) v10.2.2. Downloaded latest version 10.5.9 and discovered that this version would not allow repository editing for directing to ship resources for anti-virus updates, had to use internet to acquire updates.  Open McAfee SR# 4-20393575117 support ticket for assistance and learned that no-edit repository was by design.  McAfee was informed that their product would no longer work in our environment and they went back to their development team for re-analysis.  Beginning Expedition 378 McAfee offered version 10.6.8 which restored repository modification options.  This provides for a single location on JR for the weekly repository updates and JR MAC workstations do not have to pull their weekly updates from McAfee’s repository over the satellite internet.

• Encountered multiple MAC workstations where Adobe Acrobat would close after ~30 seconds opening. All MAC workstations exhibiting this behavior had Adobe Acrobat v10.1.1 installed.  Found a patch update v10.1.12 and installed on all affected workstations resolved matter.

Printers:

• Encountered Microscopy PC53273 not able to print, very slow responding. Further investigating revealed ITDS_Color_M855 on Suez shore printer setup on this station.  Uninstalled printer and iPrint client and reinstalled client and JR printer resolved matter.

Satellite/Internet/Phones:

• Beginning of Expedition 378 was on satellite NSS9_GA at 177° longitude and equator latitude.

• Packaged email to shore about Siem’s desire to install a UPS for the aft dome in UWGL lab. Received email response to proceed.

• Siem personnel got phone booth under stairs in Lower-Tween working.

• Jan 7: signal fade issues today, and for a period of time we were operating on Iridium until the iDirect Modem was rebooted.

o   Noticed ping latency times were noticeably faster after reboot, like 400-500ms faster.  Ping Google response now like 1100-1200ms verses 1500-1600ms yesterday.

o   Sent email request to Marlink to improve SmartWeb interface response time.  Marlink responded and implemented changes and SmartWeb interface became noticeably faster. 

• Jan 15: Observed many multiple internet outages.  Contacted Siem and was advised that they had opened a ticket with Marlink to investigate issue.  The problem was aft dome would not come online and had to maintain bow dome active which was incurring sporadic blockage from derrick.  Later, Captain changed heading and internet became solid on bow dome.  Siem still troubleshooting aft dome.

• Jan23: Internet outage for a couple hours this evening.  Started approximately 9PM and lasted thru shift change.

• Jan 27: Intermittent satellite outages during shift.  Short durations of a minute.  Multiple incidents.

• Feb 15: 2 minute internet outage at ~6AM.  Discovered later that we were running on Iridium bandwidth.  Siem notified to investigate.  Nothing conclusive discovered.

• Feb 18: Satellite outage discovered ~9AM.  Bow dome out due to derrick blockage from ship’s heading.  Aft dome out due to servo motor error code 3.  Siem has spare servomotor and proceeded with replacement.

• Feb 19: Internet services still out.  Servomotor was replaced but did not resolve satellite connection.  dB signal loss was identified as main issue per Marlink email correspondence.  However, this is due to ship’s heading and derrick blockage.

• Feb 20: JR was transitioned to new sat IS907 satellite at ~8:30AM.  Internet services were restored.

Other Equipment/Projects:

• Notified by Siem that Viji Telemedicine workstation not able to dial out across internet. Attempted multiple efforts to resolve from verifying correct protocol enabled across satellite; Sonicwall configured correctly; to bypassing firewall but all efforts failed.  However, discussions with hospital source stated that they had changed the way they do things, operates more like a video conference session such as Zoom.  New method is for JR Doctor to initiate an email/phone call for hospital services and the hospital returns with a video conference ID/code for connection.  Telemedicine dial out will no longer work.

• Downloaded multiple firmware updates for Engineering’s IMU-P ICD accelerometer device and assisted with installation. Engineer testing device and working with vendor on testing results required multiple firmware updates.

• Compiled scientists MAC workstation experiences into a report for shore management to discuss during upcoming CoChief and NSF panel meetings.

• Downloaded GMT v6.0.0 MAC & PC versions and located in their respective folders in Software volume.

• Donated two MGE 2.2 KVA UPSs to Siem from old Rignet satellite equipment.

• Performed quarterly inventory inspection as requested from shore.

• Solicited a RFQ from Power Pros for an Eaton UPS. Provided them with power consumption and pictures of our existing MGE UPS installation.  Purpose for RFQ is due to end-of-life of our existing MGE UPS by manufacturer.  Provide RFQ to Siem for review and advisement.

• Downloaded and installed the latest Google Earth Pro on the two Userroom MAC workstations per Expedition 378 scientist’s request. Placed copy of download in our Software volume.

• Siem personnel encountered a malware infestation on their network and spent weeks eradicating it. One of our Technicians aware of Siem’s travails received a suspicious email similar to what Siem encountered.  Technician inquired of MCS to investigate the email and the email was immediately reviewed and deleted.  Technician’s email diagrammed here for reference:

• AMS and checkout sheets have been update with new LTO-7 tapes inventory item.

• Assisted Engineer with utilizing installed existing Black Box Serial Server (BBSS) second port on VIT PC for new gimbalcam installation. Original a dedicated serial server was utilized but deemed not required.  Installed gimbalcam (IMENCO) software on OPS PC and VIT PC in DP for multiple gimbalcam control depending on expedition requirements.  Downloaded BBSS manuals and docs and provided to Engineer.

• Userroom sink counter was removed, resurfaced and reinstalled. Original was cracking and leaking surface water to below counter.  Looks good.

• Freight activities;

1. Worked with LO to identify items setup in AMS as freight from MCSs on e378T.

2. Printed out AMS docs for all equipment pre-setup in AMS from Exp378T.  Placed AMS forms on all existing equipment setup for freight.

3. LT51609 has a MBR Error 3 error message.  Suspect boot disk is corrupted.  Did not have Windows XP CD or USB stick to boot/fix MBR error message.  Setup laptop as AMS freight and packaged for shipment.

4. Found Dell M4400 spare parts.  Packaged and entered into AMS for freight.

5. Krakatoa server was pulled from server rack to make room for new ODAs.  Server to be surplus.  Packaged and entered into AMS for freight.

6. All LTO-5 tapes used and new are to be returned to shore per management directive. Packaged and entered into AMS for freight.

7. Pulled HP DesignJet 800ps plotter spare parts out of LTDAT.  Packaged and entered into AMS for freight.

8. Failed new HP LTO tape drive from new HP MSL6480 tape library was packaged and entered into AMS for hand carry by developer to shore.  To be returned to vendor.