If you are responsible for troubleshooting multiple machines, you need efficient ways of connecting to those machines over a network and watching the communications that goes on between them. Tools for connecting to remote computers and troubleshooting them are quite plentiful in Linux. Some of the most basic tools you need are those that let you do remote login (such as ssh), watch network traffic (such as ethereal), and those that let you open sessions and reconnect to them later (such as screen).
The ssh command is the preferred command for doing remote logins between Linux and UNIX systems. Unlike older login commands (such as telnet and rlogin), ssh creates a secure, encrypted connection between the two communicating parties. In its most simple form, you can just use ssh with the host name you want to communicate with as an argument.
# ssh myhost.linuxtoys.net root@myhost.linuxtoys.net's password: ****** Last login: Tue Jun 15 01:21:46 2004 From abc.linuxtoys.net [root@myhhost root]#
After entering the root password (represented here by stars, although you won't see anything), you should see a shell prompt. From here, you can just use the shell as though it were your local shell. A few tips about using ssh if you are new to it:
The first time you ssh to a machine, if you have not configured host authentication, ssh will remind you of that, to give you the opportunity to reject the ssh session. If you accept the connection, the RSA key fingerprint it gets from the remote machine will be used at each subsequent connection to that machine.
New Feature |
The RSA key fingerprint is stored in your $HOME/ .ssh/known_hosts file. It is possible that you may fail to connect to a remote host with ssh because the remote host changed something about its network configuration (different IP address or NIC card). You will have to remove the entry for that host from the known_hosts file to be able to ssh to the machine again. |
Both the login prompt and the title bar of the Terminal window (if you are using the default Terminal window in GNOME) will show the user you are logged in as, the remote host name, and the current directory. Make sure you remember where you are. If you forget that you are working in a remote shell, you could mistakenly reconfigure or reboot a remote server and really make a bunch of people mad.
One of the coolest features of ssh that is enabled by default on Red Hat distributions is X11 forwarding. During your ssh session, if you launch an X application from the remote host, instead of ssh trying to open that application on a desktop of the machine where the application was launched, it will open on your local desktop.
The X11 forwarding feature is done on the ssh client machine in the /etc/ssh/ssh_config file by the line ForwardX11 yes at the end of the file. If you are hooked on GUI administration tools, this feature lets you launch them remotely from server machines that may not even have an X server running.
The ssh command can also be used for remote execution. If you just want to execute a command and return to the local shell, without having to login to the remote system, you can do that on a single ssh command line. Here is an example:
# ssh myhost.linuxtoys.net date root@myhost.linuxtoys.net's password: ****** Tue Jun 15 03:11:18 CST 2004 #
As you can see, after entering the password for root on myhost.linuxtoys.net, the date command ran on the remote host. The output (the current date and time) was displayed on the local shell, the connection was closed, and the local shell prompt was returned.
To debug your ssh connection, you can add the -v option to the command line. This causes a bunch of debugging messages to be output to your screen (standard error). Using this verbose output can help you debug problems you might be having in authenticating to the remote computer or getting local configuration requests to work (such as X11 forwarding).
Ethereal is a graphical utility that lets you capture and analyze network protocol data from any of the network interfaces configured on your computer. You can watch every packet that is broadcast as it is delivered live, or capture and store the data to analyze later.
You typically start ethereal without options (type ethereal or launch it from the Red Hat menu by clicking Internet⇨More Internet Applications⇨ethereal). From the Ethereal window, click Capture⇨Start, then select the capture options you want, which includes at least the interface you want to listen on. Click OK to start gathering data and click Stop on the Capture window to stop gathering data.
There are lots of ways to configure Ethereal to capture as much or as little data as you want. You can set capture limits to a certain number of packets, kilobytes of data, or seconds. You can wait for your capture to complete before displaying the data or update the list of packets on the screen in real time.
Figure 6-4 shows an example of the Ethereal window, with capture options in the foreground.
On networks with any significant amount of traffic at all, you will probably want to do some filtering of the packets you collect. You can filter on many different fields and save the output to a file. You can also import capture files from many different network protocol analyzers, including snoop, atmsnoop, Surveyer, LANalyzer, Sniffer, Microsoft Network Monitor, and many more.
If you want to be able to login to a remote computer and leave an active session going so that you can come back to it later, you can use the screen utility. Using screen, you can detach from an active login session, then login to that computer again at another time from possibly another location and have it resume exactly where you left off.
I got this tip for using screen from Charles Mauch of the Tacoma Linux Users Group. Here's what you do:
Login to a remote computer using ssh.
Open a session with screen by simply typing:
# screen
You should now have one screen open with a shell started and ready to do your work. Change to a directory and do something like run a long make process.
To open a second screen, press Ctrl+a, c (hold the Ctrl key, press a and then c). Start another ongoing activity, such as an IRC chat session.
To get back to the first screen, press Ctrl+a, 0 (as in the number zero). Then go back to the second screen by pressing Ctrl+a, 1.
Now, to detach from this screen session press Ctrl+a, d. The [detatched] message indicates that you are now detached from your screen session.
At this point, you can log out from the remote computer completely.
When you are ready to resume your screen session, login to the machine again.
To resume your active screen sessions, type screen-r.
At this point, you can again bounce between the different screens (Ctrl+a, 0 and Ctrl+a, 1) and detach and resume from the session as you please.
Screen gives you the freedom to start a long compile or stay connected to a remote session of some sort, which you might not be able to do if you knew you were going to have to disconnect before the activity is completed. As a Linux troubleshooter, you can leave a sticky debugging problem and know that you can get back right where you were (with your current directory, command-line history, and running commands) exactly where you left them.
If you are having difficulty communicating from one system to another, here are a few basic commands that most troubleshooters will use first from a Linux system:
Run ping. Running a ping command will tell you immediately if a remote system is up and running. The time shown on each output line of ping will indicate how long a packet takes to make a round trip.
Some servers block the port that ping talks to since it can be used for denial of service attacks. In those cases, you can try using the telnet command to connect directly to a particular port that you know is accepting requests. For example, if the remote computer is a mail server, you can try using telnet to connect to the simple mail transfer protocol (SMTP) port 25. For example:
# telnet toys.linuxtoys.net 25 lrying 10.0.0.1... Connected to toys.linuxtoys.net (10.0.0.1), Escape character is ']' 220 toys.linuxtoys.net ESMTP Postfix ~.
In this example, we connected to the SMTP port (25) on toys.linuxtoys.net. We saw that we connected to the service from the line 220 toys.linuxtoys.net ESMTP Postfix. Once we knew the server was up, we could just disconnect by typing ~. on a line by itself.
If those and other network commands seem to just hang, it may be that your network connection is not active. You can try starting or restarting the network by typing:
# service network restart Shutting down interface efh0: [ OK ] Bringing up interface eth0: [ OK ]
If your Ethernet interface (eth0) doesn't come up, check for a disconnected cable, possibly a bad Ethernet card, or a misconfigured network interface. Also, check the link lights on both ends of the link.
If you can get on your network, but can't get on the Internet (or other remote network), try to ping the router or machine acting as your gateway to Internet. Then try to ping an address past that gateway.
If you can get out on the network, but you can't reach a particular host computer, try running a traceroute.
# traceroute linuxtoys.net
This will give you an idea of where your connection is not going through.
If you are getting on to the network, but are not getting a response from a computer that you know is up, try watching network traffic with ethereal or tcpdump.
In most cases, running the simple set of commands will give you a sense of where your connection to the network stands.