Question 1

What are the differences between a list, tuple, set, and dictionary in Python?

Accepted Answer

Best Answer:
List: Ordered, mutable, allows duplicates ([1, 2, 2])
Tuple: Ordered, immutable, allows duplicates ((1, 2, 2))
Set: Unordered, mutable, no duplicates ({1, 2})
Dictionary: Key-value pairs, unordered (Python 3.7+ maintains insertion order), keys must be unique ({'a': 1, 'b': 2})
Guide to Answer: Explain each with real-life use cases. Eg. List for a collection of values, Set for uniqueness, Dict for lookups, Tuple when data shouldn’t change (like coordinates).

Question 2

How do you reverse a list in Python?

Accepted Answer

Best Answer:

# Method 1
my_list[::-1]
# Method 2
my_list.reverse()
# Method 3
list(reversed(my_list))

Guide to Answer: Mention slicing for simple tasks, .reverse() when in-place is needed, and reversed() when you need an iterator. Explain when you’d prefer each.

Question 3

What is the difference between deep copy and shallow copy?

Accepted Answer

Best Answer:
Shallow copy creates a new object but inserts references to the same elements (copy.copy()).
Deep copy creates a new object and recursively copies all elements (copy.deepcopy()).
Guide to Answer: Give an example with nested lists to illustrate how shallow copy reflects changes to inner objects but deep copy doesn’t.

Question 4

How would you implement a stack or queue using Python?

Accepted Answer

Best Answer: Use list for simple needs, or collections.deque for efficient operations.
# Stack
stack = []
stack.append(1)
stack.pop()
# Queue
from collections import deque
queue = deque()
queue.append(1)
queue.popleft()
Guide to Answer: Explain time complexity: deque is O(1) for append/pop from both ends, whereas list is O(n) for pop(0).

Question 5

How do you find the most frequent element in a list?

Accepted Answer

Best Answer:
from collections import Counter
Counter(my_list).most_common(1
Guide to Answer: Mention Counter is the most Pythonic and efficient way. You could also implement manually using a dict if asked for logic.

Question 6

How do you remove duplicates from a list?

Accepted Answer

Best Answer:
# Method 1
list(set(my_list))
# Method 2 (preserving order)
list(dict.fromkeys(my_list))
Guide to Answer: Talk about whether order matters. Use dict.fromkeys() when preserving original order is needed.

Question 7

How do you sort a list of dictionaries by a specific key?

Accepted Answer

Best Answer:
sorted(data, key=lambda x: x[‘age’])
Guide to Answer: Explain lambda functions and that sorted() returns a new list, doesn’t mutate the original.

Question 8

How do you find the intersection of two lists?

Accepted Answer

Best Answer:
set(list1) & set(list2)
Guide to Answer: Use sets for better performance. Discuss when converting to set is acceptable (i.e., when order and duplicates aren’t important).

Question 9

What is the time complexity of common list operations in Python?

Accepted Answer

Best Answer:
Access by index: O(1)
Append: O(1) amortized
Insert/remove at beginning: O(n)
Search: O(n)
Delete by value: O(n)
Guide to Answer: Relate to large datasets in pipelines where performance matters. Highlight when to switch to deque or generators.

Question 10

How do you flatten a nested list in Python?

Accepted Answer

Best Answer:
# For 2D lists
[item for sublist in nested_list for item in sublist]
# With itertools
from itertools import chain
list(chain.from_iterable(nested_list))
Guide to Answer: Explain both comprehension and library-based solutions. Bonus if you mention recursion for deeply nested lists.

Question 11

How do you open and read a file in Python?

Accepted Answer

Best Answer:
with open(‘file.txt’, ‘r’) as file:
content = file.read()

Guide to Answer: Always recommend using with as it handles file closing automatically. If asked about large files, mention reading line by line using .readline() or looping.

Question 12

How do you write data to a file in Python?

Accepted Answer

Best Answer:
with open(‘output.txt’, ‘w’) as f:
f.write(“Hello, Data Engineer!”)

Guide to Answer: Discuss modes: ‘w’ (overwrite), ‘a’ (append), ‘x’ (create only if not exists), ‘b’ (binary). Clarify overwriting behavior of ‘w’.

Question 13

How do you read a large file without loading everything into memory?

Accepted Answer

Best Answer:
with open(‘bigfile.txt’, ‘r’) as file:
for line in file:
process(line)
Guide to answer:
Stress line-by-line iteration for memory efficiency. Mention chunking (read(size)) if asked for binary or custom-size chunks.

Question 14

What’s the difference between read(), readline(), and readlines()?

Accepted Answer

Best Answer:
read(): Reads entire file as one string.
readline(): Reads the next line in the file.
readlines(): Reads all lines into a list.
Guide to Answer: Use read() only when you’re sure the file is small. Use readline() or iteration for large files.

Question 15

How do you handle CSV files in Python?

Accepted Answer

Best Answer:
import csv
with open(‘data.csv’, newline=”) as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
print(row[‘column_name’])
Guide to Answer: Mention both csv.reader and csv.DictReader. Bonus if you reference pandas for structured tabular data.

Question 16

How do you handle JSON files in Python?

Accepted Answer

Best Answer:
import json
with open(‘data.json’) as f:
data = json.load(f)
Guide to Answer: Differentiate between json.load() for file objects vs json.loads() for JSON strings.

Question 17

How do you append data to a file without overwriting it?

Accepted Answer

Best Answer:
with open(‘file.txt’, ‘a’) as file:
file.write(“New line
”)
Guide to Answer: Mention using ‘a’ mode, and always use 
 if writing new lines.

Question 18

How do you handle encoding issues when reading files?

Accepted Answer

Best Answer:
with open(‘file.txt’, encoding=’utf-8′) as f:
content = f.read()
Guide to Answer: Always specify encoding when working with multilingual data. UTF-8 is a safe default.

Question 19

What happens if a file doesn’t exist and you try to open it?

Accepted Answer

Best Answer:
If mode is 'r', it raises a FileNotFoundError.
If mode is 'w' or 'a', it will create the file.
Guide to Answer: Explain exception handling using try-except for robust file processing scripts.

Question 20

How do you read and write binary files in Python?

Accepted Answer

Best Answer:
# Reading binary
with open(‘image.png’, ‘rb’) as file:
data = file.read()
# Writing binary
with open(‘copy.png’, ‘wb’) as file:
file.write(data)
Guide to Answer: Highlight the need for 'rb' and 'wb' in use cases like image processing or large binary log files.

Question 21

What is pandas, and why is it important for data engineering?

Accepted Answer

Best Answer:
Pandas is a Python library for data manipulation and analysis. It provides fast, flexible data structures like Series and DataFrame to efficiently handle structured data, making it essential for cleaning, filtering, aggregating, and transforming datasets.
💡 Guide to Answer:
Highlight how pandas simplifies handling tabular data and is a go-to tool in ETL pipelines before data is loaded into databases or analytics tools.

Question 22

What are Series and DataFrames in pandas?

Accepted Answer

Best Answer:
A Series is a one-dimensional labeled array (like a column).
A DataFrame is a two-dimensional labeled data structure with columns of potentially different types (like an Excel sheet).
💡 Guide to Answer:
Explain that DataFrame is built from Series. You can compare Series to a column and DataFrame to a table.

Question 23

How do you read data from a CSV file using pandas?

Accepted Answer

Best Answer:
import pandas as pd 
df = pd.read_csv('data.csv')
💡 Guide to Answer:
Mention parameters like sep, header, usecols, dtype, and nrows if asked for large/complex files.

Question 24

How do you filter rows in a DataFrame?

Accepted Answer

Best Answer:
df[df[‘age’] > 25]

💡 Guide to Answer:
Start with simple filters. For compound filters, use & (AND), | (OR), and enclose each condition in parentheses.

Question 25

How do you select specific columns from a DataFrame?

Accepted Answer

Best Answer:
df[[‘name’, ‘age’]]
💡 Guide to Answer:
Mention the difference between selecting one column (df['col']) and multiple columns (df[['col1', 'col2']]).

Question 26

How do you handle missing values in pandas?

Accepted Answer

Best Answer:
df.isnull().sum() # Check
df.dropna() # Drop rows with missing values
df.fillna(0) # Fill missing values
💡 Guide to Answer:
Mention that the strategy depends on context: dropping, filling with mean/median, or forward/backward filling.

Question 27

How do you change the datatype of a column?

Accepted Answer

Best Answer:
df[‘age’] = df[‘age’].astype(int)
💡Guide to Answer:
Mention common conversions like str, int, float, and datetime. Note: pd.to_datetime() for date parsing.

Question 28

How do you sort a DataFrame by a column?

Accepted Answer

Best Answer:
df.sort_values(by=’age’, ascending=False)
💡 Guide to Answer:
Include how to sort by multiple columns: by=['age', 'name'].

Question 29

How do you remove duplicate rows?

Accepted Answer

Best Answer:
df.drop_duplicates()
💡 Guide to Answer:
Use subset to target specific columns and keep='first' or 'last'.

Question 30

How do you group data in pandas and apply aggregations?

Accepted Answer

Best Answer:
df.groupby(‘department’)[‘salary’].mean()
💡 Guide to Answer:
Explain how groupby() works: split → apply → combine. Mention .agg() for multiple metrics.

Question 31

How do you merge or join two DataFrames?

Accepted Answer

Best Answer:
pd.merge(df1, df2, on=’id’, how=’inner’)
💡 Guide to Answer:
Cover how= options: 'inner', 'left', 'right', 'outer'. Explain difference from concat.

Question 32

What is the difference between merge, join, and concat?

Accepted Answer

Best Answer:
merge(): SQL-style joins on keys.
join(): Join on index or column, shorthand for merge.
concat(): Stacks DataFrames vertically or horizontally.
💡 Guide to Answer:
Use real examples: e.g., merging customer and orders tables vs stacking monthly rep

Question 33

How do you create a pivot table in pandas?

Accepted Answer

Best Answer:
df.groupby(‘department’)[‘salary’].mean()
💡 Guide to Answer:
Explain how groupby() works: split → apply → combine. Mention .agg() for multiple metrics.

Question 34

How do you apply a function to a column or row?

Accepted Answer

Best Answer:
df[‘new_col’] = df[‘salary’].apply(lambda x: x * 1.1)
💡 Guide to Answer:
Explain apply() for element-wise, applymap() for element-wise across entire DataFrame, and map() for Series-level operations.

Question 35

How do you detect and remove outliers in pandas?

Accepted Answer

Best Answer:
# Using IQR
Q1 = df[‘value’].quantile(0.25)
Q3 = df[‘value’].quantile(0.75)
IQR = Q3 – Q1
df_filtered = df[(df[‘value’] >= Q1 – 1.5 * IQR) & (df[‘value’] <= Q3 + 1.5 * IQR)]
💡 Guide to Answer:
Briefly explain the IQR rule, and why it’s more robust than Z-score for skewed data.

Question 36

What is an ETL pipeline, and how would you build one in Python?

Accepted Answer

Best Answer:
An ETL (Extract, Transform, Load) pipeline moves data from a source to a destination after applying transformations.
In Python, you can build one using scripts or frameworks. Basic example:
# Extract
data = pd.read_csv
('raw_data.csv')
# Transform
data_clean = data.dropna().rename(columns=str.lower)

# Load
data_clean.to_csv('clean_data.csv', index=False)
💡 Guide to Answer:
Mention the real-world flow: APIs → data cleaning → database or warehouse (like PostgreSQL or Snowflake). Talk about modularity and logging.

Question 37

How do you handle failures or errors in a pipeline?

Accepted Answer

Best Answer:
Use try-except blocks around each step
Implement logging to track failures
Optionally use retry logic or alerting systems
Example
try:
data = pd.read_csv(‘data.csv’)
except FileNotFoundError:
log.error(“File missing!”)
💡 Guide to Answer:
Explain your approach to making the pipeline fault-tolerant and observable.

Question 38

How do you handle duplicates or bad data in ETL?

Accepted Answer

Best Answer:
Use drop_duplicates() for exact duplicates and apply custom logic or conditions for partial duplicates. For bad data:
Validate data types
Use regex or conditionals to clean bad values
Remove or correct nulls using fillna() or dropna()
💡 Guide to Answer:
Explain that cleaning happens at the Transform stage, and tools like pandas, pydantic, or even great_expectations can be used.

Question 39

What are some common transformations in an ETL pipeline?

Accepted Answer

Best Answer:
Data type conversions
Renaming columns
Filtering or sorting rows
Aggregation or grouping
Merging with other datasets
Handling missing values
💡 Guide to Answer:
Mention how you organize your transformation logic into reusable functions or scripts.

Question 40

How do you automate a Python ETL script to run daily?

Accepted Answer

Best Answer:
Local dev: Use cron on Linux/macOS or Task Scheduler on Windows
Cloud production: Use schedulers like Apache Airflow, AWS Lambda + EventBridge, or Prefect
💡 Guide to Answer:
Talk about environment-specific options. Mention parameterization and logging for production-ready jobs.

Question 41

How do you ensure data consistency in a pipeline?

Accepted Answer

Best Answer:
Use checksums, record counts, or hashing to compare source and destination
Implement data validation checks post-load
Maintain idempotent operations when possible
💡 Guide to Answer:
Consistency is key—explain how you handle duplicates, re-runs, and broken states.

Question 42

How would you handle schema drift in a source file?

Accepted Answer

Best Answer:
Validate schema before processing
Compare expected vs actual column names
Use flexible code that can adapt, e.g., dynamic column mapping
Log differences and alert stakeholders
💡 Guide to Answer:
Schema drift is common in CSVs and APIs—show that you’re aware of the risks and can build protection.

Question 43

What’s your strategy for processing very large files or datasets?

Accepted Answer

Best Answer:
Use chunked processing in pandas (pd.read_csv(..., chunksize=10000))
Stream data line by line
If data fits in memory, use efficient types (e.g., category dtype)
Consider Spark or Dask for large-scale parallel processing
💡 Guide to Answer:
Show that you’re mindful of memory constraints and can scale when necessary.

Question 44

How do you modularize a Python ETL script?

Accepted Answer

Best Answer:
ef extract():
return pd.read_csv(‘data.csv’)
def transform(df):
return df.dropna()
def load(df):
df.to_csv(‘cleaned.csv’, index=False)
def run_etl():
df = extract()
df = transform(df)
load(df)
run_etl()
💡 Guide to Answer:
Explain how modular code improves readability, reusability, and testability.

Question 45

How do you handle time-based or incremental loads in ETL?

Accepted Answer

Best Answer:
Best Answer:
Use timestamp columns to fetch only recent records
Store state in a config file, checkpoint table, or a metadata store
Example:
# Filter new records
df[df[‘last_updated’] > last_run_time]
💡 Guide to Answer:
Highlight how incremental loads reduce load time and are more efficient for pipelines.

Question 46

What’s the difference between batch and streaming ETL?

Accepted Answer

Best Answer:
Best Answer:
Batch ETL processes data in chunks at scheduled intervals.
Streaming ETL processes data in real-time as it arrives (e.g., Kafka → Spark Streaming).
💡 Guide to Answer:
Mention that most traditional pipelines are batch; streaming is used for high-frequency event data (e.g., logs, sensors).

Question 47

How do you test your ETL pipeline?

Accepted Answer

Best Answer:
Best Answer:
Unit test each step (e.g., transformation functions)
Validate outputs (e.g., row counts, value ranges)
Use sample input files to simulate edge cases
Tools like pytest, great_expectations, or dbt for testing
💡 Guide to Answer:
Testing = confidence. Show you care about catching errors before they go downstream.

Question 48

How do you connect a SQL database using Python?

Accepted Answer

Best Answer:
import sqlite3
conn = sqlite3.connect(‘example.db’)
cursor = conn.cursor()
cursor.execute(“SELECT * FROM users”)
rows = cursor.fetchall()
conn.close()
💡 Guide to Answer:
Explain how you’d use specific connectors:
sqlite3 for local
psycopg2 for PostgreSQL
mysql-connector for MySQL
SQLAlchemy for ORM and easier scalability

Question 49

What is SQLAlchemy and why is it used in data engineering?

Accepted Answer

Best Answer:
SQLAlchemy is a Python SQL toolkit and Object-Relational Mapper (ORM) that allows developers to interact with databases using Python objects. It simplifies connection handling, query generation, and schema definitions.
💡 Guide to Answer:
Mention that it’s especially useful in large applications for maintainability and abstraction.

Question 50

How do you insert data from a pandas DataFrame into a SQL table?

Accepted Answer

Best Answer:
df.to_sql(‘table_name’, con=engine, if_exists=’replace’, index=False)
💡 Guide to Answer:
Talk about to_sql() for loading data and read_sql() for querying. Mention if_exists options: 'replace', 'append', 'fail'.

Question 51

How do you prevent SQL injection in Python?

Accepted Answer

Best Answer:
cursor.execute(“SELECT * FROM users WHERE id = ?”, (user_id,))
💡 Guide to Answer:
Explain that using string formatting is unsafe. Always use query parameters to avoid injection risks.

Question 52

What are common SQL queries a data engineer writes?

Accepted Answer

Best Answer:
SELECT, JOIN, GROUP BY, ORDER BY
INSERT, UPDATE, DELETE
CREATE TABLE, ALTER, DROP
CASE, COALESCE, WINDOW FUNCTIONS
💡 Guide to Answer:
Mention that writing optimized SQL is as important as writing Python code.

Question 53

How would you retrieve only the top 5 salaries from a table?

Accepted Answer

Best Answer:
SELECT salary FROM employees ORDER BY salary DESC LIMIT 5;
💡 Guide to Answer:
Mention use cases like leaderboards, ranking, or sampling. Add OFFSET if needed.

Question 54

How do you handle large data loads from SQL using Python?

Accepted Answer

Best Answer:
for chunk in pd.read_sql(query, con=engine, chunksize=10000):
process(chunk)
💡 Guide to Answer:
Helps when working with millions of rows. Show memory awareness and chunk processing.

Question 55

How do you handle transaction management in Python?

Accepted Answer

Best Answer:
conn = db.connect()
try:
cursor = conn.cursor()
cursor.execute(‘SOME SQL’)
conn.commit()
except Exception as e:
conn.rollback()
finally:
conn.close()
💡 Guide to Answer:
Explain commit() ensures the operation is saved; rollback() is for error handling. Especially important in batch jobs.

Question 56

How do you perform joins using pandas instead of SQL?

Accepted Answer

Best Answer:
pd.merge(df1, df2, on=’id’, how=’left’)
💡 Guide to Answer:
Mention how merge() replicates INNER, LEFT, RIGHT, and FULL joins.

Question 57

What’s the difference between relational and non-relational databases?

Accepted Answer

Best Answer:
Relational (SQL): Structured schema, uses tables (e.g., MySQL, PostgreSQL)
Non-relational (NoSQL): Flexible schema, uses collections/documents (e.g., MongoDB, Cassandra)
💡 Guide to Answer:
Give use cases: SQL for structured, transactional systems; NoSQL for flexible, high-speed apps.

Question 58

How do you connect and query a MongoDB database in Python?

Accepted Answer

Best Answer:
from pymongo import MongoClient
client = MongoClient(‘mongodb://localhost:27017/’)
db = client[‘mydb’]
collection = db[‘users’]
result = collection.find({‘age’: {‘$gt’: 25}})
💡 Guide to Answer:
Explain MongoDB structure: Database → Collection → Document. Show how find(), insert_one(), and filters work.

Question 59

How do you design a schema for analytics or reporting?

Accepted Answer

Best Answer:
Use star schema or snowflake schema
Design with fact and dimension tables
Keep columns atomic (1NF)
Consider indexes and partitioning for performance
💡 Guide to Answer:
This question tests both SQL and data warehousing knowledge. Use examples like “sales fact table joined with customer and product dimensions.”

Question 60

How do you make a GET request to an API using Python?

Accepted Answer

Best Answer:
import requests
response = requests.get(‘https://api.example.com/data’)
data = response.json()
💡 Guide to Answer:
Mention requests is the most popular HTTP library. Always check for response.status_code before parsing .json().

Question 61

How do you handle API errors or failed requests in Python?

Accepted Answer

Best Answer:
try:
response = requests.get(‘https://api.example.com/data’, timeout=10)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f”Error: {e}”)
Guide:
Talk about multi-layer validation: Salesforce, middleware, and the source system. This shows thorough planning.

Question 62

How do you pass headers and query parameters in an API call?

Accepted Answer

Best Answer:
headers = {‘Authorization’: ‘Bearer YOUR_API_KEY’}
params = {‘limit’: 100, ‘offset’: 0}
response = requests.get(‘https://api.example.com/data’, headers=headers, params=params)
💡 Guide to Answer:
Explain that headers are often used for authentication, and params help control pagination or filters.

Question 63

How do you deal with paginated API responses?

Accepted Answer

Best Answer:
while True:
response = requests.get(url, params={‘offset’: offset})
data = response.json()
process(data)

if not data[‘next’]:
break
offset += 100
💡 Guide to Answer:
Show you understand different pagination types: offset-based, cursor-based, or next-URL-based.

Question 64

How do you authenticate with an API that requires a token?

Accepted Answer

Best Answer:
headers = {‘Authorization’: ‘Bearer YOUR_ACCESS_TOKEN’}
requests.get(‘https://api.example.com/data’, headers=headers)
💡 Guide to Answer:
Talk about OAuth, bearer tokens, or API key headers. Know the difference between static keys and token exchanges.

120+ Python Interview Questions For Data Engineer

Introduction

Python interview questions for data engineer

Data Structures & Algorithms:

File Handling Questions:

Data Manipulation (pandas, numpy) Questions:

ETL Logic & Pipelines:

Database Interaction (SQL & NoSQL):

APIs & Web Scraping Questions:

Object-Oriented Programming (OOP) Questions:

Error Handling & Logging:

Concurrency & Parallelism:

Automation & Scripting:

Testing & Debugging:

Cloud & Big Data Tool Integration:

Interview questions & answers

80+ Salesforce Integration Interview Questions and Answers

Python Interview Questions for Data Engineer: Data Structures & Algorithms

Python Interview Questions for Data Engineer: File Handling

Python Interview Questions for Data Engineer: Data Manipulation (pandas, numpy)

Python Interview Questions for Data Engineer: ETL Logic & Pipelines

Python Interview Questions for Data Engineer: Database Interaction (SQL & NoSQL)

Python Interview Questions for Data Engineer: APIs & Web Scraping

Python Interview Questions for Data Engineer: Object-Oriented Programming (OOP)

Python Interview Questions for Data Engineer: Error Handling & Logging

Python Interview Questions for Data Engineer: Concurrency & Parallelism

Python Interview Questions for Data Engineer: Automation & Scripting

Python Interview Questions for Data Engineer: Testing & Debugging

Python Interview Questions for Data Engineer: Cloud & Big Data Tool Integration

Frequently Asked Questions

Conclusion

We back our work.

Your experience deserves to be read correctly.