Google Cloud: VPC Flow Logs – Analyzing Network Traffic

Big Query Cloud Cloud Logging Google Cloud Resource Monitoring
admin

Google Cloud: VPC Flow Logs – Analyzing Network Traffic

Share At:

VPC Service Controls - Logging and Monitoring Access Control on GCP Course

Overview

In this lab, we will learn how to configure a network to record traffic to and from an Apache web server using VPC Flow Logs. You will then export the logs to BigQuery for analysis.

There are multiple use cases for VPC Flow Logs. For example, you might use VPC Flow Logs to determine where your applications are being accessed from to optimize network traffic expense, to create HTTP Load Balancers to balance traffic globally, or to denylist unwanted IP addresses with Cloud Armor.

Objectives

In this lab, we will learn how to perform the following tasks:

  • Configure a custom network with VPC Flow Logs.
  • Create an Apache web server.
  • Verify that network traffic is logged.
  • Export the network traffic to BigQuery to further analyze the logs.
  • Setup VPC flow log aggregation.

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

In the Cloud Console, in the top right toolbar, click the Activate Cloud Shell button. Click Continue. it takes a few moments to provision and connect to the environment.

You can list the active account name with this command:

gcloud auth list

You can list the project ID with this command:

gcloud config list project

Task 1: Configure a custom network with VPC Flow Logs

Create the custom network

By default, VPC Flow Logs are disabled for a network. Therefore, you will create a new custom-mode network and enable VPC Flow Logs.

  1. In the Console, navigate to Navigation menu (mainmenu.png) > VPC network > VPC networks.navmenu_vpcnetworks.png
  2. Click Create VPC Network.
  3. Set the following values, leave all others at their defaults:
  1. For Subnet creation mode, click Custom.
  2. Set the following values, leave all others at their defaults:

Screenshot:

  1. Click Done, and then click Create.

Turning on VPC flow logs doesn’t affect performance, but some systems generate a large number of logs, which can increase costs.

Wait for the network to be created before proceeding to the next step.

Create the firewall rule

In order to serve HTTP and SSH traffic on the network, you need to create a firewall rule.

  1. In the left menu, click Firewall rules.
  2. Click Create Firewall Rule.
  3. Set the following values, leave all others at their defaults:

Make sure to include the /0 in the Source IP ranges to specify all networks.

  1. Click Create.

Task 2: Create an Apache web server

Create the web server

  1. In the Console, navigate to Navigation menu (mainmenu.png) > Compute Engine > VM instances.
  2. Click Create.
  3. Set the following values, leave all others at their defaults:
  1. Click Management, security, disks, networking, sole tenancy.
  2. Click Networking.
  3. For Network interfaces, click the pencil icon to edit.
  4. Set the following values, leave all others at their defaults:
  1. Click Done, and then click Create.

Install Apache

Configure the VM instance that you created as an Apache webserver and overwrite the default web page.

  1. Return to the Console, still on the VM instances page (Navigation menu (mainmenu.png) > Compute Engine > VM instances). For web-server, click SSH to launch a terminal and connect.
  2. In the web-server SSH terminal, update the package index:
sudo apt-get update
  1. Install the Apache2 package:
sudo apt-get install apache2 -y
  1. Create a new default web page by overwriting the default:
echo '<!doctype html><html><body><h1>Hello World!</h1></body></html>' | sudo tee /var/www/html/index.html
  1. Exit the SSH terminal:
exit

Task 3: Verify that network traffic is logged

Generate network traffic

  1. Return to the Console, still on the VM instances page (Navigation menu (mainmenu.png) > Compute Engine > VM instances).
  2. To view web-server, click the External IP to access the server.
This image has an empty alt attribute; its file name is image-144-1024x280.png

The Hello World! welcome page, which you configured, opens. Alternatively, you can access the server in a new tab by navigating to http://Enter the external IP Address.

Find your IP address

Find the IP address of the computer you are using. One easy way to do this is to go to a website that provides this address.

  1. Click this whatismyip.host to find your IP v4 address.
  2. Copy your IP address. It will be referred to as YOUR_IP_ADDRESS.

Access the VPC Flow Logs

  1. In the Console, navigate to Navigation menu > Logging > Logs Viewer.
  2. Under the Resources filter, select GCE Subnetwork > All subnetwork_id.
  3. Under the All logs filter, select compute.googleapis.com/vpc_flows and click OK.

If you do not see the compute.googleapis.com/vpc_flows filter option or no logs, you might have to wait a couple of minutes and refresh.

  1. Enter YOUR_IP_ADDRESS in the text search box at the top.
  1. Click on one of the log entries to expand it.
  2. Within the entry, click the arrows to expand the jsonPayload and then the connection. You may have to click Expand all to see the connection.

The connection contains the destination IP address/port number, the protocol, and the source IP address/port number. Depending on the direction of the traffic, the destination is either your web-server or the machine you are working on and vice-versa for the source. The protocol should be 6, which is the IANA protocol for TCP traffic. One of the port numbers will be 80, representing HTTP port on the web-server and the other port number should be a number greater than 1024, representing HTTP port on your machine.

Task 4: Export the network traffic to BigQuery to further analyze the logs

Create an export sink

  1. In the Console, in the left pane, click Logs Viewer.
  2. Under the Resources filter, select GCE Subnetwork -> All subnetwork_id.
  3. Under the All logs filter, select compute.googleapis.com/vpc_flows and click OK.
  4. If necessary, clear the text search box at the top and press ENTER.
  5. Click Create Sink from the top of the page.
  1. Set the following values, leave all other values at their defaults:PropertyValue (type value or select option as specified)Sink Namevpc-flowsSink ServiceBigQuerySink DestinationCreate new BigQuery dataset
  2. Name the dataset bq_vpcflows, and then click Create.
  3. Click Create Sink.
  1. Click Close when the sink is successfully created.

Generate log traffic for BigQuery

Now that the network traffic logs are exported to BigQuery, generate more traffic by accessing the web-server several times. Using Cloud Shell, you can curl the IP Address of the web-server several times.

  1. In the Console, navigate to Navigation menu (mainmenu.png) > Compute Engine > VM instances.
  2. Note the External IP address for the web-server instance to use in the next step. It will be referred to as EXTERNAL_IP.
  3. In the Cloud Shell command line, run the following command to store the EXTERNAL_IP in an environment variable. Replace the <EXTERNAL_IP> with the address you just noted.
export MY_SERVER=<EXTERNAL_IP>
  1. Access the web-server 50 times from Cloud Shell:
for ((i=1;i<=50;i++)); do curl $MY_SERVER; done

Visualize the VPC Flow Logs in BigQuery

  1. In the Console, navigate to Navigation menu (mainmenu.png) > BigQuery.
  2. If prompted, re-enter the Qwiklabs-provided student password and click Sign in.
  3. On the left-hand side, expand the bq_vpcflows dataset to reveal the table. You might have to first expand the Project ID to reveal the dataset.
  1. Click on the name of the table. It should start with compute_googleapis.

If you do not see the bq_vpcflows dataset or it does not expand, wait and refresh the page.

  1. Click on Details under the Table Details.
  1. Copy the portion of the Table ID that is after the colon(:). It will be referred to as TABLE_ID.

Table_ID in our case – bq_vpcflows.compute_googleapis_com_vpc_flows_20210822

  1. Add the following to the Query Editor and replace your_table_id with TABLE_ID while leaving the accents (`) on both sides:
#standardSQL
SELECT
jsonPayload.src_vpc.vpc_name,
SUM(CAST(jsonPayload.bytes_sent AS INT64)) AS bytes,
jsonPayload.src_vpc.subnetwork_name,
jsonPayload.connection.src_ip,
jsonPayload.connection.src_port,
jsonPayload.connection.dest_ip,
jsonPayload.connection.dest_port,
jsonPayload.connection.protocol
FROM
`your_table_id`
GROUP BY
jsonPayload.src_vpc.vpc_name,
jsonPayload.src_vpc.subnetwork_name,
jsonPayload.connection.src_ip,
jsonPayload.connection.src_port,
jsonPayload.connection.dest_ip,
jsonPayload.connection.dest_port,
jsonPayload.connection.protocol
ORDER BY
bytes DESC
LIMIT
15
  1. Click Run.

The results table shows the total bytes sent, source IP address, destination IP address, destination port, protocol, and the respective vpc name and subnet name.

If you get an error, ensure that you did not remove the #standardSQL part of the query. If it still fails, ensure that the TABLE_ID did not include the Project ID.

Analyze the VPC Flow Logs in BigQuery

The previous query gave you the same information that you saw in the Cloud Console. Now change the query to identify the top IP addresses that have exchanged traffic with your web-server.

  1. Create a new query in the Query Editor with the following and replace your_table_id with TABLE_ID while leaving the accents (`) on both sides:
#standardSQL
SELECT
jsonPayload.connection.src_ip,
jsonPayload.connection.dest_ip,
SUM(CAST(jsonPayload.bytes_sent AS INT64)) AS bytes,
jsonPayload.connection.dest_port,
jsonPayload.connection.protocol
FROM
`your_table_id`
WHERE jsonPayload.reporter = 'DEST'
GROUP BY
jsonPayload.connection.src_ip,
jsonPayload.connection.dest_ip,
jsonPayload.connection.dest_port,
jsonPayload.connection.protocol
ORDER BY
bytes DESC
LIMIT
15
  1. Click Run.

The results table now has a row for each source IP and is sorted by the highest amount of bytes sent to the web-server. The top result should reflect your Cloud Shell IP address.

Unless you accessed the web-server after creating the export sink, you will not see your machine’s IP Address in the table

Feel free to generate more traffic to the web-server from multiple sources and query the table again to determine the bytes sent to the server.

Add VPC Flow Log aggregation

You will now explore a new release of VPC flow log volume reduction. Not every packet is captured into its own log record. However, even with sampling, log record captures can be quite large.

You can balance your traffic visibility and storage cost needs by adjusting specific aspects of logs collection, which you will explore in this section.

Setting up aggregation

  1. In the Console, navigate to Navigation menu (mainmenu.png) > VPC network > VPC networks.
  2. Click vpc-net, and then click Edit.
  3. In the Subnets tab, click vpc-subnet:
  1. Click Edit > Configure logs to expose the following fields:

The purpose of each field is explained below:

  • Aggregation time interval: Sampled packets for a time interval are aggregated into a single log entry. This time interval can be 5 sec (default), 30 sec, 1 min, 5 min, 10 min, or 15 min.
  • Metadata annotations: By default, flow log entries are annotated with metadata information, such as the names of the source and destination VMs or the geographic region of external sources and destinations. This metadata annotation can be turned off to save storage space.
  • Log entry sampling: Before being written to the database, the number of logs can be sampled to reduce their number. By default, the log entry volume is scaled by 0.50 (50%), which means that half of entries are kept. You can set this from 1.0 (100%, all log entries are kept) to 0.0 (0%, no logs are kept).
  1. Set the Aggregation Interval to 30 seconds.
  2. Set the Sample rate to 25%.
  1. Click Save. You should see the following message:
50-percent.png

Setting the aggregation level to 30 seconds can reduce your flow logs size by up to 83% compared to the default aggregation interval of 5 seconds. Configuring your flow log aggregation can seriously affect your traffic visibility and storage costs.

Congratulations!

You have configured a VPC network, enabled VPC Flow Logs and created a webserver in that network. Then, you generated HTTP traffic to the webserver, viewed the traffic logs in the Cloud Console and analyzed the traffic logs in BigQuery.

Happy Learning !!!


Share At:
Website https://www.cloudblogg.com
0 0 votes
Article Rating
Subscribe
Notify of
guest
1 Comment
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
binance sign up
binance sign up
1 day ago

Your enticle helped me a lot, is there any more related content? Thanks! https://accounts.binance.com/en/register-person?ref=53551167

0
Reply

Related Posts

Back To Top

Contact Us