Simple Network Mapping using VMware tools and Netstat
Network Mapping
I had fun working out a simple way to extend application discovery using VMware tools, following a retweet or two (or 17) it seems like a lot of folks enjoyed reading about it too. So fresh from putting together a simple method to interrogate application data, helped by some very clever engineering from the VMware tools team and the function from William Lam, I thought I’d write up how we can reuse the approach along with a similar set of even simpler tools, to start mapping networks.
Knowing what servers and services are communicating and on what ports is a fundamental for troubleshooting, error detection and also decision making. For example how can we know what servers can be migrated to the cloud, if we cannot track network dependancies? How can we schedule changes?
There are many tools available that will give you a live view of your environment such as vRealize Network Insight and if you have them then you should absolutely make use of them to map your network. They are going to provide a much better level of detail and accuracy than what I’m going to show you using a combination of netstat and VMware tools. However, if you don’t have those tools, don’t have budget and a simple network map like that below appeals then read on for details on how to go about making it.
The Lab Setup
I’m going to be using Copy-VMGuestFile and Invoke-VMScript as part of the tooling to create a simple network map. These have some networking dependancies that the lab setup needs to take into consideration, I need to be able to connect from where I’m running my scripts to the ESXi hosts my targeted VMs are running on, using TCP port 902. The previous infrastructure I used was hosted in VMC, and I wasn’t about to ask if they minded exposing those ports to the public internet.
Nested is the way forward for this scenario. I’m lucky to have access to some VMC compute and had previously built a nested lab so I can use that. Under VMC I’ve a single server running AD, DNS and my PowerShell session using Windows and a 6 node vSAN cluster where I can deploy resources. I deployed 2 additional Windows servers on the nested resource to give me something to target and I opened up firewalls to allow these servers access to download patches, so I’ll have some activity to see.
The Goal
Use some very simple tooling to start building up a map of a target network. By simple I mean really simple. I don’t want to install any agents onto my Virtual Machines (outside of VMware Tools). Whilst I’m using Windows in the lab whatever is created has to be extensible to multiple operating systems. Netstat is an almost ubiquitous tool across Linux and Wintel Operating Systems. As you can see from the Wikipedia link there are a number of switches available with the command. However, I’m only interested in a couple of these;
-a Displays all active connections and the TCP and UDP ports on which the computer is listening. -n Displays active TCP connections, however, addresses and port numbers are expressed numerically and no attempt is made to determine names.
This is in part to speed up the collection of data and also because these two switches are available in both Windows and Linux.
A step by step of what I’m going to do looks like this then
- Write a simple *.bat file that captures networking information using the netstat command
- Use Copy-VMGuestFile to copy the *.bat file to the servers where we want to map the network
- Use Invoke-VMScript to execute the *.bat file
- Use Copy-VMGuestFile to gather our data output
- Reuse the *.csv merge, to bring it all together
- Manipulate the data, to what we’re interested in. Extract DNS zone information to add useful hostnames.
- Build a visualisation
Step 1 – Write a simple *.bat file that captures networking information
I’ll be targeting Windows servers in the lab so that’s why step one is to create a *.bat file, if I was targeting Linux servers then this could equally be a bash script.
netstat -an > c:\SCNetwork\%computername%.csv
Perhaps not elegant, but it is simple and It’ll get me the raw information I need to start the process. This script will execute the netstat command with the -a and -n switches and dump the output into a file named after the server it’s executed on in the SCNetwork folder. Part of the reason I’ve created this script and will go to the trouble of copying it to the endpoint is so that I can make simple use of existing environmental variables. In Windows I’m going to use ‘%computername%’ to capture the server name I’m running against, for a Linux environment this would I think in many cases mean using ‘$Hostname’.
Testing this on a local system we see the following output;
This gives me exactly what I’m looking for the protocol, local address, foreign address and connection state. This information can be used to create the network map visualisations that was setout in the goal above.
Step 2 – Use Copy-VMGuestFile to copy the *.bat file
Now I need to copy my script to where I want it to be executed. I think Copy-VMGuestFile is an under-appreciated piece of functionality that is available to use against a machine running VMware tools. This allows a suitably connected and authenticated administrator to copy files to and from a Virtual Machine running VMware Tools without having to connect over the network.
Remember earlier, I mentioned that these cmdlets make a connection directly from where they are being executed to the ESXi hosts running the Virtual Machines? Well that apparently breaks certificate trust. anyway it’s explained over on VMTN in this thread. The first 24 lines of the below script are as I understand it required to tell my session to ignore any certificate errors.
add-type @
" using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem)
{ return true; } }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$vm = Get-VM -Name NS*
Copy-VMGuestFile -source "C:\SCNetwork\netstat.bat" -Destination "C:\SCNetwork\" -Force -VM $vm -LocalToGuest -GuestUser administrator -GuestPassword VMware1!
This script is doing is copying the netstat.bat file to the endpoints I’ve identified with the $vm variable. In this example the script will only be copied to those virtual machines that start with the name ‘NS’. I’m also using the ‘force’ parameter to create the required directory and overwrite any existing files it finds.
You can see I’m also passing credentials in this script, I wouldn’t recommend doing that in a live environment, and if you don’t want to do that then the ‘GuestCredential‘ parameter will allow you to “specify a PSCredential object that contains credentials for authenticating with the guest OS where the file to be copied is located“. However, this is a lab and for indicative purposes only, therefore I’m passing credentials.
Step 3 – Use Invoke-VMScript to execute the script
Now I’ve copied the file to our target virtual machines I need to execute it. Note that we need to handle the certificate error in the same manner as before.
add-type @
" using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem)
{ return true; }
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$script = '"c:\SCNetwork\netstat.bat"'
$vm = Get-VM -Name NS*
Invoke-VMScript -ScriptType Bat -ScriptText $script -VM $vm -GuestUser administrator -GuestPassword VMware1
In this script I’m setting two variables the first to $script which is informing what I wan’t to execute, which should be no surprise to find is our copied *.bat file, and the second $vm informing our targets. again I’m passing credentials in this script, Invoke-VMScript also has a parameter ‘GuestCredential‘, which you are welcome to explore at your leisure.
All being well our *.bat file will have been copied to the target hosts and executed.
As you can see from the above we have success, my *.bat file has been copied across, the directories created, it’s been executed and I have an output *.csv file matching the hostname.
Step 4 – Use Copy-VMGuestFile to gather output
Excellent, the script has been copied and run, we now need to gather the output. We can reuse Copy-VMGuestFile for this purpose there are two parameters available to us in this cmdlet, ‘LocalToGuest’ to copy from your session to the virtual machine and ‘GuestToLocal’ to grab data from a virtual machine to your session.
add-type @
"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem)
{ return true; }
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
$vm = Get-VM -Name NS*
Copy-VMGuestFile -source "C:\SCNetwork\" -Destination "C:\SCNetwork\Data" -Force -VM $vm -GuestToLocal -GuestUser administrator -GuestPassword VMware1!
As you can see I’m passing credentials again and have to work around the aforementioned certificate issue.
The script itself is going to copy back anything it finds in the virtual machine directory ‘C:\SCNetwork’ to the location I’ve specified on my local session which is ‘C:\SCNetwork\Data’. As we’re using the force switch it will overwrite any matching files on the target side. If I was interested in interrogating the individual outputs then I may have spent a few minutes putting a timestamp into the output, but for the purposes of this worked lab I’m only really interested in the merged output, so it is enough for me to just be aware of the behaviour.
As you can see I’ve now been able to pull my captured netstat information back to a central repository.
Step 5 – Merge!
I pulled together a simple merge PowerShell script for the Application Discovery lab and this will serve our purposes perfectly for this lab.
$getFirstLine = $true
$timestamp = Get-Date -Format yy-MM-dd-hh-mm
get-childItem "C:\SCNetwork\Data" | foreach {
$filePath = $_
$lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false Add-Content "C:\SCNetwork\merged\merge-$timestamp.csv" $linesToWrite
}
As you can see this is taking the files we’ve pulled back to ‘C:\SCNetwork\Data’ and merging them together to an output file located at ‘C:\SCNetwork\merged’. I’m saving this to a different location as the script is merging the information from all the files in the source directory, if I save the file to that location I’ll have a file that grows exponentially with duplicated information.
Time stamping and saving the information to an alternate working directory adds additional value to this process as a scheduled set of tasks. That could be used to gain a more complete picture of the infrastructure. To do this all you’d need to do is modify the merge script to reflect locations and then create a ‘merge of merge’. I wouldn;t be worried about duplications as PowerQuery will allow us to deal with that very efficiently.
Step 6 – Manipulate the data, to what we’re interested in
Firstly we have another source of data that we can capture to help augment our findings. One of my servers is operating DNS, I can therefore extract the DNS zone as a *.csv, I can merge that in later on to turn foreign and local IPs into hostnames (remember we used the -n switch with netstat telling it not to try and enumerate names).
It needs to be tidied a little but this will do quite nicely.
After importing the merged netstat data we need to do a little bit of tidying, to take it from something that looks like this;
To something that looks more like this;
Before we then merge in the DNS information and create a few custom columns. I’ve created a few conditional columns that identify public address’, those that I Identified belonged to Microsoft and to allocate a colour to a few host types. I’ll explain about the colour column a little later on.
The final query I built provides an output like this;
Rather than take you step by step through the data transformation the code behind the query is below. I’m sure I could have done things in a more logical way. If this was not in a lab I’d probably spend a bit of time optimising this, but it is in a lab and working with a small data set so that’s something for another day. One thing this query code will not do is handle duplication, I’m working with a single merge file so didn’t need to do this, for those that might want to try using multiple merge sources, here is a Microsoft article for duplication removal.
let Source = Csv.Document(File.Contents("C:\Users\merge-20-01-07-11-59.csv"),[Delimiter=":", Columns=7, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}, {"Column7", type text}}),
#"Removed Top Rows" = Table.Skip(#"Changed Type",4), #"Split Column by Delimiter" = Table.SplitColumn(#"Removed Top Rows", "Column1", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, true), {"Column1.1", "Column1.2"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Column1.1", type text}, {"Column1.2", type text}}),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Changed Type1", "Column2", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, false), {"Column2.1", "Column2.2"}), #"Changed Type2" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Column2.1", Int64.Type}, {"Column2.2", type text}}),
#"Split Column by Delimiter2" = Table.SplitColumn(#"Changed Type2", "Column3", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, false), {"Column3.1", "Column3.2"}), #"Changed Type3" = Table.TransformColumnTypes(#"Split Column by Delimiter2",{{"Column3.1", type text}, {"Column3.2", type text}}),
#"Split Column by Character Transition" = Table.SplitColumn(#"Changed Type3", "Column3.2", Splitter.SplitTextByCharacterTransition((c) => not List.Contains({"0".."9"}, c), {"0".."9"}), {"Column3.2.1", "Column3.2.2"}),
#"Trimmed Text" = Table.TransformColumns(Table.TransformColumnTypes(#"Split Column by Character Transition", {{"Column2.1", type text}}, "en-GB"),{{"Column1.1", Text.Trim, type text}, {"Column1.2", Text.Trim, type text}, {"Column2.1", Text.Trim, type text}, {"Column2.2", Text.Trim, type text}, {"Column3.1", Text.Trim, type text}, {"Column3.2.1", Text.Trim, type text}, {"Column3.2.2", Text.Trim, type text}}),
#"Removed Columns" = Table.RemoveColumns(#"Trimmed Text",{"Column4", "Column5", "Column6", "Column7"}), #"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Column1.1", "Protocol"}, {"Column1.2", "Local Address"}, {"Column2.1", "Local Port"}, {"Column2.2", "Foreign Address"}, {"Column3.1", "Foreign Port"}, {"Column3.2.1", "State"}, {"Column3.2.2", "PID"}}),
#"Removed Columns2" = Table.RemoveColumns(#"Renamed Columns",{"PID"}), #"Merged Queries" = Table.NestedJoin(#"Removed Columns2", {"Local Address"}, simonlocal_dnszone, {"Data"}, "simonlocal_dnszone", JoinKind.LeftOuter),
#"Filtered Rows" = Table.SelectRows(#"Merged Queries", each ([State] = "CLOSE_WAIT" or [State] = "ESTABLISHED")), #"Expanded simonlocal_dnszone" = Table.ExpandTableColumn(#"Filtered Rows", "simonlocal_dnszone", {"Name", "Data"}, {"simonlocal_dnszone.Name", "simonlocal_dnszone.Data"}),
#"Renamed Columns1" = Table.RenameColumns(#"Expanded simonlocal_dnszone",{{"simonlocal_dnszone.Name", "Local Name"}}), #"Merged Queries1" = Table.NestedJoin(#"Renamed Columns1", {"Foreign Address"}, simonlocal_dnszone, {"Data"}, "simonlocal_dnszone", JoinKind.LeftOuter),
#"Expanded simonlocal_dnszone1" = Table.ExpandTableColumn(#"Merged Queries1", "simonlocal_dnszone", {"Name", "Data"}, {"simonlocal_dnszone.Name", "simonlocal_dnszone.Data.1"}),
#"Renamed Columns2" = Table.RenameColumns(#"Expanded simonlocal_dnszone1",{{"simonlocal_dnszone.Name", "Foreign Name"}}),
#"Replaced Value" = Table.ReplaceValue(#"Renamed Columns2",null,"Unknown",Replacer.ReplaceValue,{"Foreign Name"}),
#"Added Conditional Column" = Table.AddColumn(#"Replaced Value", "Foreign Host Final", each if [Foreign Name] <> "Unknown" then [Foreign Name] else if Text.StartsWith([Foreign Address], "51.105.249") then "Microsoft_Public" else if not Text.StartsWith([Foreign Address], "192") then "Public" else [Foreign Name]),
#"Filtered Rows1" = Table.SelectRows(#"Added Conditional Column", each ([Foreign Address] <> "127.0.0.1")),
#"Removed Columns1" = Table.RemoveColumns(#"Filtered Rows1",{"simonlocal_dnszone.Data", "simonlocal_dnszone.Data.1"}),
#"Added Conditional Column1" = Table.AddColumn(#"Removed Columns1", "Colour", each if [Foreign Host Final] = "Unknown" then "#ff0000" else if [Foreign Host Final] = "Public" then "#85cfde" else if Text.StartsWith([Foreign Host Final], "Microsoft") then "#b8ff33" else null)
in
#"Added Conditional Column1"
Reference: Now that this query is built, we don’t have to build it again. To manipulate a new data set in the same way we can either edit the source line above to reference a new merge file or from within the GUI we can step back through the applied steps to the source element and make the change there, before replaying.
Step 7 – Build a visualisation
Now comes the fun part and we need to bring the data to life through visualisation. I’m going to work with a couple of custom visualisations available from Microsoft AppSource or GitHub. Firstly Network Navigator Chart and secondly Attribute Slicer.
Network Navigator in it’s own words; “Network Navigator lets you explore node-link data by panning over and zooming into a force-directed node layout (which can be precomputed or animated live). From an initial overview of all nodes, you can use simple text search to enlarge matching nodes in ways that guide subsequent navigation. Network nodes can also be color-coded based on additional attributes of the dataset and filtered by linked visuals.”
Attribute Slicer in it’s own words; “Attribute Slicer lets you filter a dataset on a given column by selecting attribute values of interest. The initial display is a helpful overview that lists the most common values first and shows the overall distribution of values as a horizontal bar chart. Whenever you select an attribute value, it is moved to the list of applied filters and all records containing that value are added to the result set for further analysis.”
These two visualisations are going to for the basis for how we’re going to present the netstat data. Both of these resources have documentation that describes how they can be configured, which can do it far better than I can, however I will provide a little detail for how I’m using them.
I’m using the Attribute Slicer as a traditional data slicer, I found it a while back liked the way it functioned and if I think a slicer will add value to a visualisation this is the one I include.
I’m using Network Navigator to represent the relationships we’ve discovered between servers within our netstat data set. The first and arguably most important elements of this visualisation are the ‘Source Node‘ and ‘Target Node‘ fields, these are mapped to the ‘Local Name‘ and ‘Foreign Host Final‘ respectively. Secondly there is also a field for ‘Target Node Colour‘ this is where we can use the custom column, ‘Colour‘, I used to map a #HTML colour codes. I used these to flag connections flagged as unknown in red, those to public endpoints in blue and that which was identified as belonging to Microsoft in green.
Results and Findings
I ran the script against just 3 servers, NS-TEST-01 and 02 and SC-STEP-01. I posted a teaser snippet of the results earlier on in this post, here is the full output.
The various elements are as follows; the main visualisation is reserved for Network Navigator showing the relationships between nodes, 5 slicers on the right hand side for data interrogation the last section displays the raw data.
You can see there are elements that we have not interrogated that are being represented on the visualisation, such as the ESX hosts, virtual centre server, public addresses and a flag for something unknown. These are being displayed because they are foreign hosts in the netstat data. So even though I only ran the script against 3 servers I’m seeing 11 servers represented in my visualisation. We’re seeing that the data is representing and building out the relationships between servers.
As part of the query built out earlier, I created a conditional column that returned set values based on the ‘Foreign Address’ data. I know the network I was running this against, there was a single internal address space 192.168.200.0/24, anything inside of that should be in DNS anything outside of that public and I shouldn’t have any unknowns being flagged. This conditional column took any foreign address that started with ‘51.105.249’ and identified that as ‘Microsoft_Public’ and anything that did not start with ‘192’ and flag that as public, if neither of these conditions were met it returned the ‘Foreign Name’ field. What that should have given me was a data set with either the DNS name, Microsoft_Public or Public and nothing else. However, I’ve got an ‘Unknown’ being flagged.
By clicking on the unknown node in the Network Navigator view, all the other information dynamically updates. Useful in allowing me to track down what it is and why it’s flagged as unknown. looking at the data I can see the foreign port is listed as ‘3389’, which is for RDP connectivity. From that port I know a couple of things 1) it’s a windows server and 2) it’s accepting connections on RDP. So for starters in trying to track this down, I can just make an RDP connection to the server…
Sure enough I could connect to the offending host, and discover that this was a rush job Veeam server that was added to the lab at the back of last year to allow us to replicate the lab to a new site. (I’ll let Dean Lewis blog about that!). We were able to use the visualisation to provide meaningful information in one click that told me more about the network than I knew before, pretty cool.
Summary
Whilst I was writing this, I could think of even more ways that we could extend or apply the same principals to gathering data.
For example we could pull and merge the arp table from the servers we interrogate. That way we could use the local address from netstat data and merge that with the mac address from the arp table, this would allow identification and association of hardware vendor with hostname/address. The arp table will also include information for gateways and firewalls, devices that are normally connected through, providing a mechanism to reflecting this information in the visuals.
A second example, although this would need more work, might be executing a ‘tasklist’ command on each endpoint and extended the netstat to include the PID. We could then match the PID from each network connection to the application. That would give another method of slicing the data. Although an easier way might be to merge in a data set that matches TCP ports to applications and services!
Even with just this level of rudimentary network dependancy information, more informed decisions can be made about the impact of changes being made in the environment, about the impact of migrating certain servers. This has been completed without installing a single agent, using simple tools available on most operating system distributions and off the shelf visualisation tools. I’d say that was goal met.
Thanks
Simon