Azure Vnet Flow Logs Overwriting the file

Hello Nisha Das

Can you please update us if the action plan provided was helpful?

Should there be any follow-up questions or concerns, please let us know and we shall try to address them.

Please do consider to “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Hi @Ganesh Patapati thanks for sharing this.

I investigated the scenario and created a script to read the logs, but the solution doesn’t seem to resolve the issue. Below is the script I used.

import datetime
from azure.storage.blob import BlobServiceClient
from urllib.parse import urlparse

def get_vnet_flow_log_data(connection_string, blob_path):
    # Initialize the Blob Client
    parsed = urlparse(blob_path)
    path_parts = parsed.path.lstrip('/').split('/', 1)
    container_name = path_parts[0]
    blob_name = path_parts[1] if len(path_parts) > 1 else ""
    blob_service_client = BlobServiceClient.from_connection_string(connection_string)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)

    # 1. Get the Block List
    # This retrieves the metadata about the "chunks" that make up the file
    block_list = blob_client.get_block_list(block_list_type='all')
    print(f"Block List Type: {type(block_list)}")
    print(f"Block List: {block_list}")
    
    # get_block_list returns a tuple of (committed_blocks, uncommitted_blocks)
    committed_blocks = block_list[0]  # First element is committed blocks
    print(f"Total Blocks in Blob: {len(committed_blocks)}")

    results = []
    current_offset = 0

    # 2. Iterate through blocks and read ranges
    i = 1;
    for block in committed_blocks:
        block_size = block['size']
        
        # Download only the specific range of bytes for this block
        # This avoids downloading the entire massive log file
        stream = blob_client.download_blob(offset=current_offset, length=block_size)
        block_content = stream.readall().decode('utf-8')
        print(f"Block {i}: Size={block_size}, Content={block_content[:]}")  # Print first 100 chars of each block
        i += 1

        
        results.append(block_content)
        current_offset += block_size

    return results

# --- Configuration ---

CONN_STR = ""



# Path format: flowLogResourceID=/SUBSCRIPTION_ID_NETWORKWATCHERRG/.../y=2023/m=07/d=09/h=03/m=00/macAddress=0022485D8CF8/PT1H.json
BLOB_URL = ""

try:
    log_blocks = get_vnet_flow_log_data(CONN_STR, BLOB_URL)
    
    # Print the second-to-last block (the latest log entry before the closing bracket)
    if len(log_blocks) >= 2:
        print("Latest Flow Log Entry:")
        print(log_blocks[-2])
except Exception as e:
    import traceback
    print(f"Error: {e}")
    traceback.print_exc()

To test this, I started the VM and generated some traffic. When I ran the script, the output showed 10 block blobs, which is expected.
Next, I stopped the VM, started it again, and generated additional traffic. After running the script again, it returned only 5 block blobs covering the time interval from 11:41 to 11:43. However, the block blobs from 11:23 to 11:30 are missing.
My concern is this: if I read the file at 11:42, it appears I can only retrieve logs from 11:41 to 11:43, and the data from 11:23 to 11:30 is lost. Moreover, I wouldn’t even be aware that this data was lost because the file seems to have been overwritten by the time it’s read.

Could you please let me know how to prevent potential data loss in this scenario? FYI: This is only happening when we stop the VM. If the VM runs continuously, it is working as expected.

Hi @Ganesh Patapati could you please advise on this.

Hello Nisha Das

Thanks for the reply!

• I have started a private message with you. Please review the private chat and share the required details so we can proceed with investigating the issue.

Hi @Ganesh Patapati , is there any update on this? In our last call, you mentioned that you would create a document outlining this behavior or would share this with your team. Could you please let me know the current status?