Last Activity: 7 December 2014, 8:24 PM EST

Thanks Given: 1

hey guys..........i am currently working on shell script that resolves status code 58 in netbackuptool .............this error is actually coz when server cannot communicate to the client so i am trying to write a shell script to do this automatically instead of series of steps manually..............the troubleshooting are as follows.........
When troubleshooting status 58 errors on a NetBackup client, the first thing to test is to whether or not you can access the client from the client Host Properties from the master server. If that works the same ports are involved when backing up the client as to when you access the client host properties. Connecting to the client host properties from the master server would prove the master server can access the client without any problems. So if using a storage unit on the master server you should be able to backup the client without getting the status 58 error.
In 6.x environments you can use the command- bptestbpcd to verify you can connect to both the vnetd and bpcd ports on the client server.
e.g.
bptestbpcd -verbose -debug -client <client hostname>
If still getting status 58 errors after ensuring the master server can connect to the client, focus on the media server used during the backup. The problem is more than likely with the client hostname lookup or gethostbyname call made to DNS or local host file on the media server.
If the master server can not connect to the client when trying to access the client host properties, or the point of failure appears to be with the media server, below are steps you can take to further troubleshoot this issue-
To test the master/media server resolution of the client server hostname run the following command-
<install path>/netbackup/bin/./bpclntcmd -hn <client hostname>
Since reverse lookups is part of the NBU server to client connections make sure the client can also be resolved by its IP address-
<install path>/netbackup/bin/./bpclntcmd -ip <client IP address>
On the client test the resolution of the NBU servers by issuing the same commands. These commands should be run against all of the media servers that may be trying to backup the client server-
<install path>/netbackup/bin/./bpclntcmd -hn <NBU server hostname>
<install path>/netbackup/bin/./bpclntcmd -ip <NBU server IP address>
Verify you are able to 'ping' the client's IP address from the NBU server. If this fails consult with your Network Administrator and client server System Administrator to resolve the layer 3 or IP network connectivity. Double check the server's NIC's IP address and netmask to ensure they are configured correctly.
A very useful command to help with testing client and server resolution is the command -
bpclntcmd -pn when run from a NBU client or media server.
What that command does when ran on a client as example,
1. The client gets the first server listed in it's server's list and knowing it is the master server does a forward lookup of that hostname or a gethostbyname call to DNS or host file.
2. Once that is successful you should see the message-'expecting response from (master server hostname)'. If the forward lookup fails or the client does not have the bprd port 13720 defined in the registry for Windows clients or in /etc/services for UNIX clients you will not see that message displayed. Create a bplist log on the client at verbose 5 and rerun that command.
3. When the client connects to the bprd port on the master server, the master server performs two functions. It first does a simple reverse lookup of the incoming IP address and that is the first name returned by the command. Then it goes into the policy database and lookup what hostname it is using when backing up the client server. That is the second hostname returned by the command. See the example below-
In this example the client hostname is dotto.veritas.com and the master server is hal9000.veritas.com-
C:Program FilesVERITASNetBackupbin>bpclntcmd -pn
expecting response from server hal9000.veritas.com
dotto.veritas.com dotto.veritas.com 10.82.110.6
3412
Creating a bplist log on the client at verbose 5 should show the attempt to resolve the master server and make a connection request to the NBU server.
Creating a bprd log on the master server should show the incoming connection as well.
The same bprd log when create on the master server would show the attempt to connect to the client using the client host properties.
Resolution:
The next step if the client is still failing with a status 58 after you verified the servers are resolvable to each other using the bpclntcmd command, is to ensure the client server has the bpcd port in listening mode-
For Windows clients-
netstat -a and look for the BPCD and VNETD port are listening-
e.g.
TCP dotto:vnetd dotto.veritas.com:0 LISTENING
TCP dotto:bpcd dotto.veritas.com:0 LISTENING
The netstat -an command can also be useful in determining if a TCP SYN request is making it to the client server.
After issuing the command from the Netbackup server -bptestbpcd -client (hostname) if you then on the client server run the command-
netstat -an | grep 13724
You should see a incoming TCP SYN request going to the client's vnetd port 13724 as a SYN_RECV.
e.g.
tcp 0 0 192.168.70.205:4480 192.168.70.198:13724 SYN_RECV
That would show the netbackup server is sending the TCP SYN request and it is being seen on the client side. It can also verify the correct source IP address is being used by the Netbackup server. If the source IP address does not appear to be the correct one the OS is choosing, especially if the Netbackup has a directly connected IP segment, check for the Netbackup server using the REQUIRE_INTERFACE setting. That setting forces all outbound TCP SYN requests to use the source IP address defined by the REQUIRED_INTERFACE setting and can not be used in environments where you have the requirements to run backups over multiple network segment.
For UNIX client verify the ports are listening by issuing the same command-
# netstat -a |grep bpcd
*.bpcd *.* 0 0 49152 0 LISTEN
#netstat -a |grep vnetd
*.vnetd *.* 0 0 49152 0 LISTEN
For Windows clients if it not listening on the ports verify bpinetd is running on the client using task mgr.
For UNIX clients verify BPCD and VNETD are defined in /etc/service and inetd-
# vi /etc/services
bpcd 13782/tcp bpcd
vnetd 13724/tcp vnetd
# vi /etc/inetd.conf
bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd
vnetd stream tcp nowait root /usr/openv/bin/vnetd vnetd
For UNIX client server if the /etc/services file for bpcd shows it is using tcpd then there is a TCP wrapper on the bpcd port 13782-
This example shows a TCP wrapper running on the bpcd port when defined in inetd.conf-
bpcd stream tcp nowait root /usr/local/bin/tcpd /usr/openv/netbackup/bin/bpcd bpcd
UNIX (Linux) - check for any firewall settings. e.g. service iptables status
Locally on the client create a bpcd log and increase the verbose level to 5.
On the client server to test whether the bpcd binary is executable and will generate a log issue the following command-
UNIX- <install path>netbackup/bin/./bpcd
Windows- <install path>program filesVERITASnetbackupbin:bpcd
That should generate the following log entry and prove the binary is executable and generates a log entry-
BPCD log. You should see something like this -
16:47:45.986 [5296.3376] <2> bpcd main: offset to GMT 21600
16:47:45.986 [5296.3376] <2> bpcd main: Got socket for input 3
16:47:46.017 [5296.3376] <2> logconnections: getsockname(3) failed: 10038
16:47:46.017 [5296.3376] <16> bpcd setup_sockopts: setsockopt 1 failed: h_errno 10038
16:47:46.017 [5296.3376] <2> bpcd main: setup_sockopts complete
16:47:46.158 [5296.3376] <2> vauth_acceptor: ..libvlibsvauth_comm.c.332: Function failed: 17 0x00000011
16:47:46.158 [5296.3376] <16> bpcd main: authentication failed: 17
It is normal to have it end with the <16> bpcd main: authentication failed: 17 error. But this test proves the binary is functioning correctly.
Then to test the bpcd port locally on the client server from the command prompt, run this telnet test using the loopback interface-
telnet localhost 13782 or
telnet 127.0.0.1 13782
That telnet to the loopback interface using the localhost hostname should generate a log that looks like this-
16:49:35.352 [3336.4360] <2> bpcd main: offset to GMT 21600
16:49:35.352 [3336.4360] <2> bpcd main: Got socket for input 376
16:49:35.352 [3336.4360] <2> logconnections: BPCD ACCEPT FROM 127.0.0.1.3845 TO 127.0.0.1.13782
16:49:35.352 [3336.4360] <2> bpcd main: setup_sockopts complete
16:49:35.414 [3336.4360] <2> bpcd peer_hostname: Connection from host localhost (127.0.0.1) port 3845
16:49:35.414 [3336.4360] <2> bpcd valid_server: comparing hal9000.veritas.com and localhost
16:49:35.414 [3336.4360] <4> bpcd valid_server: localhost is not a master server
16:49:35.414 [3336.4360] <16> bpcd valid_server: localhost is not a media server either
16:49:39.189 [3336.4360] <16> bpcd main: read failed: The operation completed successfully.
It generates the <16> error because it does not understand what telnet is and also fails to authenticate the localhost as a NBU server.
For Linux clients if they are missing a library file required by bpcd or vnetd you would get this type of error message-
telnet localhost bpcd
Trying 127.0.0.1...
Connected to clientname.domainname.com
Escape character is '^]'.
bpcd: error while loading shared libraries: libstdc++-libc6.2-2.so.3:
cannot open shared object file: No such file or directory
Contact the OS vendor to obtain the required library file.
If the telnet to localhost works try the same telnet test from the master server using the client hostname to the bpcd port-
telnet (client hostname) 13782
That should generate a log entry as seen below showing the master server being accepted-
16:52:46.077 [1160.5436] <2> bpcd main: offset to GMT 21600
16:52:46.077 [1160.5436] <2> bpcd main: Got socket for input 400
**The client sees the incoming connection from the master server using the IP address 10.82.105.254**
16:52:46.077 [1160.5436] <2> logconnections: BPCD ACCEPT FROM 10.82.105.254.44554 TO 10.82.110.6.13782
16:52:46.077 [1160.5436] <2> bpcd main: setup_sockopts complete
**Performs a reverse lookup of the incoming IP address and gets the hostname**
16:52:46.092 [1160.5436] <2> bpcd peer_hostname: Connection from host hal9000.veritas.com (10.82.105.254) port 44554
**Then compares the hostname to the server list on the client**
16:52:46.092 [1160.5436] <2> bpcd valid_server: comparing hal9000.veritas.com and hal9000.veritas.com
**The hostname compare succeeds**
16:52:46.092 [1160.5436] <4> bpcd valid_server: hostname comparison succeeded
16:52:49.476 [1160.5436] <16> bpcd main: read failed: The operation completed successfully.
If any of these telnet tests fails to generate a log entry then there is something outside of NBU that is preventing access to the client's port. Most likely firewall software or a TCP wrapper was placed on the ports. Checking what services the client is running for Windows clients (Try Turning off MS firewall software for a quick test) or check the hosts.allow and hosts.deny files on the UNIX clients would be a good place to review next. Those files typically are in the /etc directory as in /etc/hosts.allow.
Windows 2008 clients may have the Domain Firewall, Private Firewall, and Public Firewall turned on. Either disable these firewalls in Windows firewall properties. Or create exceptions for ports 13724, 13782, and exceptions, if necessary for Netbackup processes.
For UNIX clients you can try removing the BPCD port from inetd.conf and run the command 'bpcd -standalone' to see if that gets the port listening.
Note: When testing connections to bpcd note the delta time difference between the bpcd log message e.g. '16:52:46.077 [1160.5436] <2> logconnections: BPCD ACCEPT FROM 10.82.105.254.44554 TO 10.82.110.6.13782 ' and the log message '16:52:46.092 [1160.5436] <4> bpcd valid_server: hostname comparison succeeded' Due to the hard coded bprbm timeout of 1 minute this time difference can not exceed 60 seconds. If it exceeds that timeout then try using a host file entry to avoid DNS latency.
If the client had been working at NBU 5.x release but now fails after upgrading to NBU 6.x, you can try using 5.x defaults for the client connection to see if that gets the failing client working again. To do so, from the Administration Console:
1. Launch the NetBackup Administration Console, connecting to the master server
2. Expand Host Properties in the left pane
3. Select Master Server in the left pane
4. Click the name of the master server in the right pane
5. Select the Client Attributes section
6. Add the name of the client in question if it isn't listed
7. In the Connect Options tab for the client, make the following changes:
BPCD connect back -> 'Random port'
Ports -> 'Reserved port'
Daemon connection port -> 'Daemon port only'
i am trying to create this script only for unix clients but the problem ii am not able to frame the logic and stuck how to start................please help.