This course equips systems engineering graduate students with the statistical and computational tools needed to analyze data rigorously, make defensible inferences, and build reproducible analytical workflows.

Data science sits at the intersection of statistics, computer science, and domain expertise. It transforms raw data into actionable knowledge through systematic analysis.

The standard format for statistical analysis: rows = observations, columns = variables/features.

Random sampling means every individual in the population has an equal (or known) probability of being selected. This ensures sample statistics are unbiased estimators of population parameters.

Reproducibility means that given the same data and code, anyone should be able to arrive at the same results.

# Standard imports — copy this block into every notebook
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import statsmodels.api as sm
import warnings
warnings.filterwarnings('ignore')

# Reproducibility
np.random.seed(42)

# Plot settings
plt.rcParams['figure.dpi'] = 100
plt.rcParams['figure.figsize'] = (8, 4)
sns.set_style('whitegrid')

print('Environment ready.')

# Array creation
a = np.array([10, 20, 30, 40, 50])
b = np.arange(0, 1, 0.1)        # [0.0, 0.1, ..., 0.9]
c = np.linspace(0, 1, 11)       # 11 evenly spaced points
d = np.zeros(5)                  # [0, 0, 0, 0, 0]

# Element-wise operations (no loops!)
print(a * 2)                     # [20, 40, 60, 80, 100]
print(a + b[:5])                 # Element-wise addition

# Descriptive statistics
print(f"Mean: {a.mean()}, Std: {a.std(ddof=1)}")
print(f"Min: {a.min()}, Max: {a.max()}")

# Boolean indexing
above_25 = a[a > 25]            # array([30, 40, 50])

np.random.seed(42)

# Normal distribution: 1000 samples, mean=100, std=15
samples = np.random.normal(loc=100, scale=15, size=1000)
print(f"Mean: {samples.mean():.2f}, Std: {samples.std():.2f}")

# Uniform distribution
uniform = np.random.uniform(low=0, high=1, size=500)

# Exponential (inter-arrival times, λ=15/hr)
arrivals = np.random.exponential(scale=1/15, size=1000)

# 2D arrays (matrices)
X = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])
print(X.shape)        # (3, 3)
print(X[:, 1])        # Column 1: [2, 5, 8]
print(X.mean(axis=0)) # Column means: [4, 5, 6]

# From a dictionary
data = {'Machine': ['CNC-01', 'CNC-02', 'OVEN-01'],
        'MTBF': [720, 540, 1200],
        'Utilization': [0.82, 0.91, 0.65]}
df = pd.DataFrame(data)

# From a CSV file (local or URL)
url = 'https://raw.githubusercontent.com/gedeck/practical-statistics-for-data-scientists/master/data/state.csv'
state = pd.read_csv(url)

# From a NumPy array
arr = np.random.randn(100, 3)
df_arr = pd.DataFrame(arr, columns=['feature_1', 'feature_2', 'feature_3'])

df.head()              # First 5 rows
df.tail(3)             # Last 3 rows
df.shape               # (rows, columns)
df.dtypes              # Data type of each column
df.info()              # Types + non-null counts + memory
df.describe()          # Summary stats (count, mean, std, quartiles)
df.isnull().sum()      # Missing value count per column
df.nunique()           # Unique value count per column
df.columns.tolist()    # Column names as a list

# .loc — label-based indexing
df.loc[0:2, 'Machine']             # Rows 0-2, column 'Machine'
df.loc[df['MTBF'] > 600, :]        # Filter + all columns

# .iloc — position-based indexing
df.iloc[0:2, 1:3]                  # Rows 0-1, columns at index 1-2
df.iloc[-1, :]                     # Last row

# Single column
df['MTBF']                         # Returns a Series
df[['Machine', 'MTBF']]            # Returns a DataFrame

# Boolean masking (filtering)
high_util = df[df['Utilization'] > 0.80]
combined = df[(df['MTBF'] > 500) & (df['Utilization'] < 0.90)]

# Arithmetic on existing columns
df['Availability'] = df['MTBF'] / (df['MTBF'] + 24)

# Conditional column with np.where
df['Risk'] = np.where(df['Utilization'] > 0.85, 'High', 'Normal')

# Apply a custom function
df['MTBF_Category'] = df['MTBF'].apply(
    lambda x: 'Excellent' if x > 1000 else 'Good' if x > 500 else 'Poor'
)

# Binning continuous variables
df['Util_Bin'] = pd.cut(df['Utilization'], bins=[0, 0.5, 0.8, 1.0],
                         labels=['Low', 'Medium', 'High'])

# Detection
df.isnull().sum()                  # Count per column
df.isnull().mean() * 100           # Percentage per column

# Deletion
df_clean = df.dropna()             # Drop rows with any NaN
df_clean = df.dropna(subset=['MTBF'])  # Drop only if MTBF is NaN

# Imputation
df['MTBF'].fillna(df['MTBF'].median(), inplace=True)  # Median fill
df['Risk'].fillna('Unknown', inplace=True)              # Category fill

# Basic groupby
df.groupby('Type')['MTBF'].mean()

# Multiple aggregations
summary = df.groupby('Type').agg(
    Avg_MTBF=('MTBF', 'mean'),
    Std_MTBF=('MTBF', 'std'),
    Count=('Machine', 'count')
).round(1)

# Groupby + filter
reliable = df.groupby('Type').filter(lambda g: g['MTBF'].mean() > 600)

# Groupby + transform (broadcast back to original shape)
df['MTBF_zscore'] = df.groupby('Type')['MTBF'].transform(
    lambda x: (x - x.mean()) / x.std()
)

fig, axes = plt.subplots(2, 2, figsize=(10, 8))

# Histogram — distribution shape
axes[0,0].hist(data, bins=30, edgecolor='white')
axes[0,0].set_title('Histogram')

# Boxplot — quartiles and outliers
axes[0,1].boxplot(data)
axes[0,1].set_title('Boxplot')

# Scatter — bivariate relationship
axes[1,0].scatter(x, y, alpha=0.5)
axes[1,0].set_title('Scatter Plot')

# Bar — categorical comparison
axes[1,1].bar(categories, values)
axes[1,1].set_title('Bar Chart')

plt.tight_layout()
plt.show()

# Distribution plot with KDE
sns.histplot(data=df, x='MTBF', kde=True)

# Boxplot by category
sns.boxplot(data=df, x='Type', y='MTBF')

# Violin plot — richer than boxplot
sns.violinplot(data=df, x='Type', y='MTBF', inner='quartile')

# Pairwise scatter matrix
sns.pairplot(df[['MTBF', 'Utilization', 'Defect_Rate']])

# Correlation heatmap
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', vmin=-1, vmax=1)

# FacetGrid — same plot across categories
g = sns.FacetGrid(df, col='Type', col_wrap=2)
g.map(plt.hist, 'MTBF')

# 1. Load
url = 'https://raw.githubusercontent.com/.../state.csv'
state = pd.read_csv(url)

# 2. Inspect
print(state.shape, state.dtypes)
print(state.describe())

# 3. Clean
state = state.dropna()

# 4. Engineer
state['Pop_Density'] = state['Population'] / state['Area']

# 5. Summarize by group
state.groupby('Region')['Murder.Rate'].agg(['mean','std','count'])

# 6. Visualize
fig, ax = plt.subplots(1, 2, figsize=(12, 4))
state['Murder.Rate'].hist(bins=15, ax=ax[0])
state.boxplot(column='Murder.Rate', by='Region', ax=ax[1])
plt.tight_layout()
plt.show()

Multimodal AI tools (ChatGPT, Claude, Gemini) can help interpret complex statistical visualizations.

Open the Week 1 Colab notebook and complete the following steps:

Save your notebook to Drive: /SEN608/W1/. Submit the sharing link via Moodle → SEN608 → Week 1 → Assignment.

Complete the following tasks in your Colab notebook:

AI-Assisted Exploration (3rd Edition)

1. Use an AI tool to learn about robust estimates of location. Explore implementations in Python.
2. Improve the data dictionary of a public dataset (e.g., UCI ML Repository) using AI. Compare with the original.
3. Have AI interpret one of your visualizations. Deepen with follow-up questions. Critically evaluate the output.

Deadline: before Week 2 class. Submission: Moodle Assignment module. Late penalty: -20% per day (max 3 days).

Preparation:

• If your Python experience is limited: complete the Python Fundamentals notebook from the course page.
• Read Bruce et al., Chapter 1 (Exploratory Data Analysis) carefully.
• Complete Assignment 1 and submit via Moodle.
• Prepare at least one methodological question about the dataset you analyzed.