Masa's blog

Fujitsu BX900: How to save MMB System Event Log remotely

I know MMB SEL can be saved using Web GUI, or log in to MMB via SSH and follow the menu. These methods require several mouse clicks or keyboard interactions. It would be nice to automate this process using a script or one-liner commands, wouldn't it?

Here are some examples.

CIM (This method requires WS-MAN enabled in MMB)

$ wbemcli ei 'http://<mmb user name>:<password>@<mmb IP address>:5988/root/cimv2:Linux_MgmtBladeLog' |awk -F[,\"] '{print sprintf("%04d", $30), $27,$24}'

$ wbemcli ei 'http://admin:admin@192.168.10.10:5988/root/cimv2:Linux_MgmtBladeLog' | awk -F[,\"] '{print sprintf("%04d", $30), $27,$24}'
0001 2016-07-19 10:44:44 The IPv4 NTP server 2 is recovered.
0002 2016-07-19 10:19:09 The IPv4 NTP server 2 is inaccessible.
0003 2016-06-09 15:07:27 The IPv4 NTP server 2 is recovered.
0004 2016-06-09 15:01:57 The IPv4 NTP server 2 is inaccessible.
0005 2016-06-06 15:08:17 Server Blade-10 was removed.
0006 2016-06-06 15:06:55 Server Blade-10 power on.
0007 2016-06-06 15:06:04 Server Blade-10 power off.
0008 2016-06-06 15:01:49 Server Blade-10 power on.
0009 2016-06-06 15:01:01 Server Blade-10 power on.
0010 2016-06-06 15:00:53 Server Blade-10 was added.
...

SSH

Create a file containing the following:

$ echo -e "7\n2\n0\n0\n5" > input_file

Execute SSH using the file created as input. plink can be downloaded from PuTTY download page.

$ ssh -T <mmb user name>@<mmb IP address> < input_file

$ plink -ssh -T -l <mmb user name> -pw <password> <mmb IP address> < input_file

$ plink -ssh -T -l admin -pw admin 192.168.10.10 < input_file
...
SEL entry number:511, Free space of SEL entry:0
2016-07-19 10:44:44 Info/Normal The IPv4 NTP server 2 is recovered.
2016-07-19 10:19:09  Minor      The IPv4 NTP server 2 is inaccessible.
2016-06-09 15:07:27 Info/Normal The IPv4 NTP server 2 is recovered.
2016-06-09 15:01:57  Minor      The IPv4 NTP server 2 is inaccessible.
2016-06-06 15:08:17 Info/Normal Server Blade-10 was removed.
2016-06-06 15:06:55 Info/Normal Server Blade-10 power on.
2016-06-06 15:06:04 Info/Normal Server Blade-10 power off.
...

If needed, you can redirect the output to a file to save the contents.

Also you can use the above SSH method to save MMB System Information Dump remotely.The difference is that the input file needs to contain the following:

Or just run

$ echo -e "7\n5\n0\n0\n5" > input_file

Then execute ssh or plink against MMB.

Example)

$ plink -ssh -T -l admin -pw admin 192.168.10.10 < input_file
...
+----------------------------------------------+
+             System Control Information
+----------------------------------------------+
System Name        : BX900-2
Housing Type       : BX900S1
System Location    : Data Center
System Contact     : dc-admin@example.com
Chassis Part Number: A3C40119934


+----------------------------------------------+
+             MMB
+----------------------------------------------+
Management Blade Run Mode             : master
Management Blade Status               : ok
Management Blade Manufacture          : FUJITSU
Management Blade Manufacture Date     : 04/02/2010 10:17:00
...

EFI Shell on Fujitsu PRIMERGY servers

Fujitsu removed the EFI shell access from their BIOS images some time ago for PRIMERGY servers (I believe Ivy Bridge generation or later models), so you cannot bring up EFI Shell in UEFI mode. In normal cases, this is not a problem. However in some cases, EFI shell makes your life easier.
For example, if the OS boot entry is somehow lost, without EFI Shell, you probably need to boot off of a Linux live CD or something to recreate a boot entry. If EFI shell is available, you don't need to do that. You just simple bring up the shell, and mount the EFI partition to execute the OS boot image.
Another example is that if you want to update PCIe card firmware image such as RAID card firmware before installing an OS, you have to use Fujitsu's Update Manager Express to do the update. It will take long time as it will update the whole system (BIOS, iRMC FW, PCIe FW, etc.). If you have EFI Shell, you can run EFI utilities to update firmware, and it will take less time.
Although Fujitsu disabled EFI Shell, there is still a way to use EFI shell on PRIMERGY servers.

Here are the instructions.

Download the EFI shell binary from tianocore.
Format your USB flash drive in FAT32.
Create the \efi\boot folder.
Copy the downloaded binary to \efi\boot, and rename the file to bootx64.efi.
Boot the PRIMERGY server off of the USB device in UEFI mode.

You can extract the USB EFI shell environment as a HDD image, and mount it via iRMC Remote Storage. It's handy.

I put EFI utilities like LSI's storcli, Intel's bootutil, etc. to this, and created a HDD image as "efitools.img", so I can flash Fujitsu RAID firmware (ROM file), or change the boot mode (PXE or iSCSI) for Intel-based network cards on any PRIMERGY servers.

Fujitsu ServerView Agents and ServerView RAID Manager for Ubuntu

Fujitsu ServerView Agents is a server management agent for Fujitsu PRIMERGY servers, and Fujitsu ServerView RAID Manager is a RAID management utility for RAID controllers in PRIMERGY. The latest ServerView Agents and RAID Manager for Ubuntu published on Fujitsu download site are old, and does not support the latest platforms. I have ported SV Agents 7.20.28 for Linux and SV RAID Manager 6.4.3 for Linux to Ubuntu 14.04 and 16.04 and created deb packages to manage Fujitsu PRIMERGY servers running Ubuntu.

Here are the packages I ported.

serverview-raid-manager_6.4-3_amd64.deb
serverview-connectorservice_2.10.03-01_amd64.deb
serverview-eecd_7.20-28_amd64.deb
serverview-agents_7.20-28_amd64.deb
serverview-ssmwebui_7.20-28_all.deb

I am not authorized to distribute these packages, so if you need deb packages, please ask Fujitsu support. Fujitsu may have the latest packages. I just could not find them on the Fujitsu site.

Here are some screenshots.

ServerView System Monitor Web GUI

ServerView RAID Manager

ServerView Agents status check command output (srvmagt status)

# srvmagt status

Installation script for ServerView Agents Linux v7.20-28
(c) Fujitsu Technology Solutions 2016

Getting status of ServerView Agents Linux ...
               Server model: PRIMERGY BX2580 M2
                   Hostname: bx900s1-u-b8 (192.168.64.148)
           Operating system: Ubuntu 16.04 LTS [UBUNTU1604]
               Linux kernel: 4.4.0-28-generic on x86_64, SELinux unknown, EDAC enabled, target system
            ServerView eecd: serverview-eecd    7.20-28
  ServerView SNMP subagents: serverview-agents  7.20-28
      System Monitor Web UI: serverview-ssmwebui        7.20-28
ServerView Remote Connector: serverview-connectorservice        2.10.03-01
            ServerView Raid: serverview-raid-manager    6.4-3
               SNMP service: snmpd      5.7.3+dfsg-1ubuntu4

    ServerView module smbus: exists, loaded, /dev/pci/smbus exists
        Native IPMI modules: requested, loaded, /dev/ipmi0 exists

        SNMP daemon (snmpd): running (1629), UDP #161 OK, AgentX socket OK SNMP communication OK
     RAID daemon (amDaemon): running (1646), TCP 3173 OK
     ServerView eecd daemon: running (40798), /dev/eecd[as] OK
          SVRemoteConnector: running (17877), TCP 3172 OK
           ServerView scagt: running (42222), status OK (1)
          ServerView sc2agt: running (42234), status OK (1)
          ServerView busagt: running (42246), status OK (1)
           ServerView hdagt: running (42256), status OK (1)
         ServerView unixagt: running (42266), status OK (1)
        ServerView etheragt: running (42276), status OK (1)
         ServerView biosagt: running (42286), status OK (1)
        ServerView securagt: running (42298), status OK (1)
       ServerView statusagt: running (42308), status OK (1)
          ServerView invagt: running (42318), status OK (1)
          ServerView thragt: running (42329), status OK (1)
           ServerView vvagt: running (42340), status OK (1)
        ServerView hpsimagt: running (42350), status OK (1)
          ServerView vmeagt: running (42394), status OK (1)
     ServerView svupdateagt: running (42360), status OK (1)
           ServerView osagt: running (42370), status OK (1)
      ServerView vmesnmpagt: running (42417), status OK (1)
Getting status completed with result 0 (OK status).

vCenter Server Event Log is Filled Up with Log in/Log out Events from Fujitsu ServerView Operations Manager

If you monitor Fujitsu PRIMERGY servers running VMware ESXi with Fujitsu ServerView Operations Manager (SVOM), you probably notice that there are many log in/log out events from your SVOM server recorded in the event log (see the following screenshot for example):

By default, SVOM checks the host status through Fujitsu ServerView CIM provider which is included in Fujitsu Custom ESXi 5.x/6.0 images every five minutes (300 seconds). That is why you see the log in/log out events in every 5 minutes. In case you want to see some important events/messages in the event log like when a failure occurs, these unimportant login/logout events make you harder to find what you are looking for. You can change the monitoring interval or even disable the polling. To do this, follow the procedure described below.

Log in to your SVOM.
Open up a ServerList window.
Select the host where you want to change the interval, and bring up the Server Properties window.
Go to the Network/SNMP tab, and change the interval in Poll Interval. In this example, I changed it to 600 seconds (10 minutes). If you set this value to 0, that means the polling will be disabled for this host. Once you confirm the value, click the OK button to apply the change.

If you are using Fujitsu PRIMERGY servers and all are running ESXi, it would be better to use ServerView plugin for vCenter Server. With the plugin, you can monitor all your PRIMERGY server hardware status within vCenter Server. Also you can launch iRMC which is baseboard management controller on PRIMERGY (similar to DELL iDRAC, or HP iLO), open a remote KVM, and update BIOS and firmware.

You can download the latest plugin (as of 3/2/2016, it's V2.2.8) from the Fujitsu download site. Here is the direct link to the download page.

FUJITSU Software ServerView Plug-in for VMware vCenter V2.2.8

Fujitsu ServerView Operations Manager showing Link Up/Link Down alerts in every one or two minutes for Windows Server

When you manage your Fujitsu PRIMERGY servers running Windows Server with ServerView Agents, you sometimes see a lot of "Link Up/Link Down" alerts in ServerView Operations Manager like the following screenshot:

In this case, check what interface is reporting the link problem. In the above example, it shows "Communication link failed at the station 15.", so the interface index you should check is 15. In most cases, it is "Teredo Tunneling Interface".

Log in to the Windows Server system, and bring up a command prompt. Run the following commands to find out the interface:

C:\> netsh interface ipv4 show interfaces
C:\> netsh interface ipv6 show interfaces

In the above, the interface index 15 is "Teredo Tunneling Pseudo-Interface", so if you don't need this, disable it.

C:\> netsh interface teredo set state disable

This should stop the frequent up/down events in SVOM. If you see this for an interface other than "Teredo Tunneling Pseudo Interface", you probably have a real connection issue.

IPMITOOL on VMware ESXi

The IPMITOOL command is a great tool to configure the local BMC network, get the sensor/FRU information, etc. I use this command a lot on my Linux systems. Since virtualized servers (mostly VMware ESXi) are increasing, I want to use the ipmitool command in ESXi.
There are lots of blog posts talking about creating an ipmitool binary for ESXi out there. I would like to take a (little) further step to integrate the tool to my standard ESXi image so that I can deploy ESXi to my servers with the ipmitool command integrated.
Here is what I did.

Create an ipmitool binary. See the following blogs for details:
- ESXi ipmitool(works on any Linux system as well)
- Make Ipmitool Working For ESXi 5.1
Create an ipmitool vib. I used VIB Author on my Linux system (CentOS 6.7 x86_64). If needed, you can download the ipmitool vib I created from here (ipmitool-1.8.15-1.vib).
Create a custom ISO (and an offline bundle) with the ipmitool vib. Read the vSphere manuals on how to create a custom ISO/offline bundle using Image Builder. This VMware blog entry would help:
- Using the vSphere ESXi Image Builder CLI
Deploy the new image to servers.

I used the VIB I created in step 2 with ESXi 5.5 U3 and ESXi 6.0 U1, and it worked on both versions.

Here are some outputs.

~ # vmware -vl
VMware ESXi 5.5.0 build-3248547
VMware ESXi 5.5.0 Update 3
~ # esxcli software vib list |grep ipmitool
ipmitool                       1.8.15-1                               IPMItool  CommunitySupported  2016-01-14
~ # /opt/ipmitool/bin/ipmitool mc info
Device ID                 : 52
Device Revision           : 2
Firmware Revision         : 1.00
IPMI Version              : 2.0
Manufacturer ID           : 10368
Manufacturer Name         : Fujitsu Siemens
Product ID                : 1062 (0x0426)
Product Name              : Unknown (0x426)
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    IPMB Event Generator
    Chassis Device
Aux Firmware Rev Info     :
    0x07
    0x52
    0x00
    0x46
~ # /opt/ipmitool/bin/ipmitool lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD OEM
Auth Type Enable        : Callback : MD5 PASSWORD OEM
                        : User     : MD5 PASSWORD OEM
                        : Operator : MD5 PASSWORD OEM
                        : Admin    : MD5 PASSWORD OEM
                        : OEM      : MD5 PASSWORD OEM
IP Address Source       : BIOS Assigned Address
IP Address              : 192.168.14.37
Subnet Mask             : 255.255.255.0
MAC Address             : 90:1b:0e:80:70:a9
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 10.0 seconds
Default Gateway IP      : 192.168.14.1
Default Gateway MAC     : 00:00:0c:07:ac:40
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max   : XaaaaaaaaXXaXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
~ # /opt/ipmitool/bin/ipmitool fru print 2
 Chassis Type          : Blade
 Chassis Part Number   : ABN:K1467-V200-54
 Chassis Serial        : YLSR000000
 Chassis Extra         : BX2580M1
 Chassis Extra         : FUJITSU
 Board Mfg Date        : Mon Dec  8 22:02:00 2014
 Board Mfg             : FUJITSU
 Board Product         : D3321
 Board Serial          : SQ1449MA00000
 Board Part Number     : A3C40161777
 Board Extra           : D3A
 Board Extra           : 02
 Board Extra           : BX2580 M1
 Product Manufacturer  : FUJITSU
 Product Name          : PRIMERGY BX2580 M1
 Product Part Number   : ABN:K1467-V200-54
 Product Version       : C
 Product Serial        : YLSR000000
 Product Asset Tag     : 02
 Product Extra         : a26799
 Product Extra         : 0426

Update on 02/26/2016: I changed the link to the VIB package. You should be able to download it without asking me for access permissions.

How to turn off the "Invalid or Missing Entitlement Certificates" message on RHEL

If your RHEL system is not registered with Red Hat Network, you will see the warning message, "Invalid or Missing Entitlement Certificates. This system is missing one or more valid entitlement certificates.", at the top right corner after you log in to the system at run level 5.

Of course, the message will be automatically turned off after you register your system with Red Hat. However if you just want to suppress the message without registering the system, that is possible with a few simple steps. What you need is to disable the rhsm-icon process. Here is the procedure.

Log in to your RHEL system.
Open a terminal, and change the current directory to /etc/xdg/autostart

$ cd /etc/xdg/autostart

Rename the rhsm-icon.desktop file.

$ mv rhsm-icon.desktop rhsm-icon.desktop.bak

Log out and log back in to the system

Now you don't see the warning message :)