content.json

{"pages":[{"title":"schedule","text":"","link":"/schedule/"}],"posts":[{"title":"System Design","text":"Demystifying Web Infrastructure and Technologies: A Comprehensive GuideIntroductionFrom our everyday interactions with various web applications to the seamless flow of data across platforms, the internet’s vast landscape thrives on numerous technologies working in harmony. As a technology enthusiast, I embarked on a quest to better understand this intricate fabric of interconnected technologies. In this blog, I aim to unravel my findings and offer you a peek into the enigmatic realms of the internet, covering a wide range of topics from the basic to the advanced. The following tutorial video. Vertical ScalingVertical Scaling, also known as “scaling up,” is a strategy that involves augmenting the capacity of a single server, often through the addition of resources like CPU, RAM, or storage. For instance, if a web server struggles with traffic, one might choose to upgrade its processor or increase its memory to improve performance. However, vertical scaling has its limitations: there’s a finite amount of resources you can add to a single server. Horizontal ScalingIn contrast, Horizontal Scaling or “scaling out” involves adding more servers to distribute the load. It’s akin to adding more lanes on a highway to handle increased traffic. Horizontal scaling can provide enhanced redundancy and handle more requests concurrently. Services like Amazon’s Elastic Load Balancer enable automatic horizontal scaling. Load BalancersLoad Balancers play a vital role in both horizontal and vertical scaling. They distribute network traffic across multiple servers to ensure no single server bears an overwhelming load. This distribution enhances application responsiveness, increases availability and reliability, and can dynamically accommodate fluctuating demand. Content Delivery NetworksContent Delivery Networks (CDNs) optimize delivery of web content to users based on their geographical location. CDNs consist of a network of servers distributed globally, serving cached content closer to the user’s location, leading to quicker load times and less bandwidth consumption. CachingCaching is the practice of storing copies of data in high-speed access areas to serve future requests faster. It’s like keeping a snapshot of frequently accessed data so that the system can avoid re-fetching or recomputing it. This process is integral to improving application speed and performance. IP AddressIP Address, short for Internet Protocol Address, is a unique identifier assigned to each device on a network. It facilitates the routing of packets across the internet, similar to how a postal address directs mail to your home. TCP/IPTCP/IP (Transmission Control Protocol/Internet Protocol) is the suite of communication protocols that the internet and most networking architectures use. It facilitates reliable, ordered, and error-checked delivery of data packets between applications running on hosts in an IP network. Domain Name SystemDomain Name System (DNS) translates human-friendly domain names (like www.example.com) into machine-readable IP addresses. It’s akin to a phonebook for the internet, ensuring we don’t have to memorize complex IP addresses. HTTPHTTP (Hypertext Transfer Protocol) is a request-response protocol in client-server computing models. It is the foundation of data communication on the World Wide Web, transmitting hypertext messages between clients and servers. RESTREST (Representational State Transfer) is an architectural style for designing networked applications. It uses HTTP methods (GET, POST, PUT, DELETE) to make calls between machines. It’s stateless and lightweight, making it a popular choice for APIs. GraphQLGraphQL is a query language for APIs, enabling clients to request exactly what they need, reducing over-fetching or under-fetching of data. It empowers the client to dictate the structure of the response, resulting in more efficient network requests. gRPCgRPC (Google Remote Procedure Call) is a high-performance, open-source framework that uses protocol buffers (protobufs) for serializing structured data. gRPC supports multiple programming languages, is highly scalable, and can run in any environment. WebSocketsWebSockets facilitate full-duplex communication channels over a single TCP connection. Unlike HTTP, where the client must initiate communication, WebSockets provide two-way communication between client and server, making them ideal for real-time applications like gaming or chat applications. SQLSQL (Structured Query Language) is a standard language for managing and manipulating relational databases. SQL’s ACID (Atomicity, Consistency, Isolation, Durability) properties ensure reliable processing of database transactions, which is essential for data integrity. NoSQLNoSQL databases emerged to address the limitations of SQL databases, particularly for scaling and for handling unstructured data. They offer flexible schemas and are designed to scale horizontally across servers. ShardingSharding is a type of database partitioning that separates large databases into smaller, faster, more easily managed parts called data shards. Each shard is held on a separate database server instance, spreading the load and reducing the risk of a single point of failure. ReplicationReplication is the process of sharing information across multiple databases to improve reliability, fault-tolerance, or accessibility. It provides a mechanism to ensure data remains consistent across distributed systems. CAP TheoremCAP Theorem is a concept that a distributed computing system cannot simultaneously provide all three of the following guarantees: Consistency (all nodes see the same data), Availability (every request receives a response), and Partition Tolerance (the system continues to operate despite network failures). Message QueuesMessage Queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. They are essential for managing process and thread communication and coordination in complex systems. In summary, the technologies and concepts we’ve covered are fundamental to how the web works, driving our day-to-day interactions online. Understanding these intricacies fosters an appreciation for the complexities that power the digital world and gives us the knowledge to create efficient, reliable systems of our own. I hope that this blog helped demystify these concepts and has sparked your curiosity to dive deeper.","link":"/2023/07/10/20concepts/"},{"title":"Theoretical Basics of Greedy Algorithms","text":"By choosing the local optimum at every stage, we aim to achieve the global optimum. The key to selecting a greedy algorithm is: it’s possible to deduce the global optimum from the local optimum. How to verify if you can use greedy: Provide a counterexample: If you can’t think of a counterexample, try harder. Mathematical induction. Steps for a greedy algorithm: Decompose the problem into several sub-problems. Identify an appropriate greedy strategy. Find the optimal solution for each sub-problem. Stack the local optimums to form a global optimum. Example: Leetcode 455: Distributing Cookies Local Optimum: Give the largest cookie to the child with the biggest appetite. Global Optimum: Feed as many children as possible. 12345678910111213141516171819202122232425class Solution: def sort(self, nums): for i in range(len(nums) - 1): for j in range(i + 1, len(nums)): if nums[j] < nums[i]: temp = nums[i] nums[i] = nums[j] nums[j] = temp def findContentChildren(self, g, s): self.sort(g) self.sort(s) count = 0 i = 0 # pointer to g j = 0 # pointer to s while i < len(g) and j < len(s): if s[j] - g[i] >= 0: count += 1 i += 1 j += 1 else: j += 1 return count Example: Leetcode 376 Swing SequenceThis question is divided into three situations: there are flat slopes in the upper and lower slopes; only the first and last elements; monotonic slopes in which there are flat slopes; (prediff just records the direction of the initial slope when the swing appears)Assume that the rightmost side has a slope, so the initial RESULT = 1.12345678910111213141516171819class Solution: def wiggleMaxLength(self, nums): # only one element if len(nums) == 1: return 1 # three conditions: Up and down with flat slopes; first element; monotone with flat slopes result = 1 prediff = 0 # This is equivalent to adding another element at the beginning of the array with the same value as its first element. curdiff = 0 for i in range(len(nums) - 1): curdiff = nums[i + 1] - nums[i] if (prediff <= 0 and curdiff > 0) or (prediff >= 0 and curdiff < 0): result += 1 # Satisfies the need for a flat slope up and down # Dealing with monotony with flat slopes prediff = curdiff return result Example: Leetcode 53 Maximum Subarray SumIf successive sums are negative, they are immediately discarded and the next number is chosen as the starting position,using greedy algorithm:123456789101112131415class Solution: def maxSubArray(self, nums): result = -1000000 # Set result to a very small value count = 0 # Record the current traversed value for i in range(len(nums)): count += nums[i] if count > result: result = count if count < 0: count = 0 # Reset count to start accumulating from the next position, reflecting the greedy approach return result Example: Leetcode 860 Lemonade To Find ChangeGreedy thought: leave as many 5’s as possible because 5’s are more versatile and can make change not only to 20 but also to 10, whereas 10 can only make change to 20. 12345678910111213141516171819202122232425class Solution: def lemonadeChange(self, bills): five = 0 ten = 0 twenty = 0 for i in range(len(bills)): if bills[i] == 5: five += 1 if bills[i] == 10: if five == 0: return False else: five -= 1 ten += 1 if bills[i] == 20: if ten > 0 and five > 0: ten -= 1 five -= 1 elif five >= 3: five -= 3 else: return False return True","link":"/2023/08/10/Greedy%20Algorithm/"},{"title":"MoreCAP","text":"Diving Deeper into the CAP TheoremAs a fresh graduate exploring the vast landscape of Computer Science, I find myself particularly intrigued by principles that serve as the backbone of the digital world. The CAP Theorem, known as Brewer’s Theorem, is one such principle that provides fundamental guidelines when designing distributed systems. In this blog post, I will try to shed some light on this intricate topic in a comprehensible manner. What is CAP Theorem?The CAP Theorem is a concept in distributed computing that states a distributed data store cannot simultaneously provide all three of the following guarantees: Consistency: Every read receives the most recent write or an error. Availability: Every request receives a response, without guarantee that it contains the most recent write. Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes. In simpler terms, when designing distributed systems, we can only guarantee two out of these three properties at any given time. This trade-off has significant implications on the design and usability of distributed systems. Exploring the Trade-offsTo understand these trade-offs better, let’s discuss each of the three guarantees and their implications: Consistency: This means that all nodes in the system see the same data at the same time. Achieving high consistency often requires a sacrifice in availability, especially during network partitions. Highly consistent systems ensure that once a write operation is confirmed, all subsequent read operations will reflect that write. Availability: This means that the system is always ready to process requests and provide responses, irrespective of the state of some individual nodes in the system. Highly available systems ensure that they provide a response to every request, but there’s a chance that some of these responses might not reflect the most recent writes, especially in the case of network partitions. Partition Tolerance: In a distributed system, nodes are often spread across different networks and data centers, which makes network failures inevitable. Partition tolerance means that the system continues to function and uphold its guarantees even when network failures occur. In such cases, depending on whether the system prioritizes consistency or availability, it will either delay responses until the partition is resolved (favouring consistency), or continue to process requests independently on both sides of the partition (favouring availability). Practical Implications of the CAP TheoremThe CAP Theorem guides the design of distributed systems and helps engineers understand the trade-offs that they have to make. For instance: A system like Apache Cassandra opts for Availability and Partition Tolerance (AP), offering eventual consistency where data might not immediately be the same across all nodes but will eventually become consistent. Conversely, a system like Google’s Spanner opts for Consistency and Partition Tolerance (CP), ensuring all nodes have a consistent view of data at the cost of availability during network partitions. It’s essential to remember that these trade-offs depend on the system’s use-case and the application’s specific needs. There’s no one-size-fits-all solution, and system designers often need to make careful considerations about which guarantees to prioritize. In conclusion, the CAP theorem is a fundamental principle in distributed systems design. Although it presents a trade-off situation, understanding these trade-offs allows us to build better, more robust distributed systems. As a new graduate delving deeper into distributed systems, the CAP theorem has been a critical guidepost, illuminating the landscape of design decisions and their implications.","link":"/2023/07/20/MoreCAP/"},{"title":"This is my First Post","text":"Very happy!Today I build up my blog!Main purpose of my blog are Record the my experience of learning computer science. Record my life. I think we can’t study without life since we live in such a beautiful world!Life is not only for study, but also happy, sad, family, friendship and love. I’ll be glad if someone read my blog and get some help.","link":"/2022/01/24/My-New-Post/"},{"title":"System Design","text":"Operating SystemsI’m learning system design and want to post a overview about operating system based on my understanding: File Systems, Virtual Memory, Memory Paging, and Instruction Execution CycleTable of Contents Introduction File Systems Virtual Memory Memory Paging Instruction Execution Cycle Conclusion Introduction An Operating System (OS) is the most crucial program that runs on a computer. It manages the computer’s memory, processes, and all of its software and hardware. It also facilitates communication between the user and the hardware components of the system. Today, we will delve deeper into four key aspects of an OS: File Systems, Virtual Memory, Memory Paging, and the Instruction Execution Cycle. File Systems A file system is a method and data structure that the OS uses to manage files on a disk or partition. It controls how data is stored and retrieved on a disk. Types of file systems include FAT32, NTFS, HFS+, ext4, etc. Each has different features, performance, security, and capacity handling characteristics. I’ve done a File system project in my operating system class. Virtual Memory Virtual memory is a memory management technique used by the OS which gives an application the impression that it has a large, contiguous working memory, while in fact, it might be fragmented and may overflow onto disk storage. It makes the application independent of the physical memory’s actual size. Memory Paging Paging is a memory management scheme implemented by the OS to enable the physical memory’s efficient utilization. It allows the physical address space of a process to be non-contiguous, breaking physical memory into fixed-size blocks called frames and logical memory into blocks of the same size called pages. Instruction Execution Cycle The Instruction Execution Cycle, also known as the fetch-decode-execute cycle, is the basic operational process of a computer. It is the process by which a computer retrieves a program instruction from its memory, determines what actions the instruction requires, and carries out those actions. Conclusion Operating systems are integral to computing. They manage various tasks to allow users to interact seamlessly with their hardware. Understanding the key aspects of an OS such as file systems, virtual memory, memory paging, and the instruction execution cycle can help users and developers leverage the OS for optimized performance.","link":"/2023/07/05/OperatingSystem/"},{"title":"Northwestern University","text":"Today I receive a Master of computer science offer from Northwestern University!!! I’m proud to take the offer and go to this Top10 university in United States.","link":"/2022/10/24/offer/"},{"title":"Database Indexing","text":"Yesterday I had an interview with the question about indexing, I didn’t answer very well so I’d write a blog to record my study and understanding for future useage.Understanding the Need for Database Indexes: A Simple ExampleThrough a straightforward example, this tutorial aims to elucidate the necessity of database indexes. Let’s consider we possess a database table named Employee, encompassing three columns: Employee_Name, Employee_Age, and Employee_Address. Imagine this Employee table hosts thousands of rows of data.Suppose we wish to retrieve information on all employees named ‘Jesus’ from this table. We opt to utilize the following query statement: 1SELECT * FROM Employee WHERE Employee_Name = 'Jesus' What transpires if the table lacks an index?Upon executing this query, what unfolds as the database endeavors to locate employees named Jesus? The database is compelled to sift through each row within the Employee table to ascertain whether the employee’s name (Employee_Name) matches ‘Jesus’. Since our goal is to acquire information on every employee named Jesus, halting the search after finding the first matching row is not an option, as there could be additional rows meeting the criteria. Thus, the search must continue row by row until the final one is examined - indicating the database must inspect thousands of rows to identify all employees named Jesus. This process is known as a full table scan.How Do Database Indexes Enhance Performance?One might ponder if conducting a full table scan for such a rudimentary task seems inefficient - shouldn’t the database be more astute? This is akin to manually scanning an entire table from start to finish - slow and far from elegant (“not at all sleek”). However, as you might infer from the title, this is precisely where indexes come into play. The essence of using indexes is to expedite search operations by reducing the number of records/rows in a table that need to be examined. What is an Index?An index is a data structure that stores the values of a specific column in a table (most commonly, a B-Tree). Indexes are created on the columns of a table. Hence, a key takeaway is that an index contains the values of a column in a table, and these values are stored within a data structure. Remember, an index is a data structure.Which Data Structures Can Serve as Indexes?B-Trees are the most frequently utilized data structures for indexes because they offer low time complexity for search, deletion, and insertion operations, which can all be performed in logarithmic time. Another critical reason is that the data stored in B-Trees is ordered. The choice of data structure for indexes is usually made by the Relational Database Management System (RDBMS), but in some cases, you can specify the data structure to use when creating an index.How Do Hash Table Indexes Work?Hash tables are another type of data structure you might encounter as indexes - often referred to as hash indexes. The rationale for using hash indexes is their high efficiency in locating values. Hence, for queries that involve comparing strings for equality, such as the previously discussed query (SELECT * FROM Employee WHERE Employee_Name = ‘Jesus’), a hash index on the Employee_Name column can significantly speed up value retrieval. The operation of hash indexes involves using the column’s values as index keys, with the keys’ corresponding actual values being pointers to the relevant rows in the table. Since hash tables can essentially be viewed as associative arrays, a typical data item might resemble “Jesus => 0x28939”, where 0x28939 is a reference to the memory location of the row containing Jesus. Querying a value like “Jesus” in a hash index and obtaining a reference to the corresponding row in memory is evidently much faster than scanning the entire table to find rows with the value “Jesus”.Drawbacks of Hash IndexesHash tables are unordered data structures, rendering them ineffective for many types of query statements. For example, if you sought to find all employees under 40 years of age, how would you use a hash index for the query? It’s unfeasible because hash tables are suited for key-value pair queries - i.e., queries for exact matches (such as “WHERE name = ‘Jesus’”). The key-value mapping of hash tables also implies that their storage of keys is unordered. This is why hash indexes are typically not the default data structure for database indexes - they lack the flexibility offered by B-Trees.Are There Other Types of Indexes?Indexes using R-Trees as their data structure are typically utilized for spatial queries. For instance, a query like “Find all Starbucks within 2 kilometers of my location” would benefit from a database table indexed with an R-Tree, enhancing query efficiency. Another type of index is the bitmap index, which is well-suited for columns containing boolean values (true and false), especially when many instances of these values (representing true or false) are present - essentially, columns with low selectivity.How Do Indexes Improve Performance?Since indexes essentially serve to store the values of a column in a data structure, they facilitate quicker retrieval of these column values. If the index uses the most common data structure, a B-Tree, then the data within it is ordered. Having ordered column values can significantly boost performance. Here’s why:Assume we create a B-Tree index on the Employee_Name column. This means that when we perform the aforementioned SQL search for employees named ‘Jesus’, we no longer need to scan the entire table. Instead, we utilize the index to locate employees named ‘Jesus’ because the index has already sorted the names alphabetically. The sorted nature of the index implies that querying a name becomes much faster, as employees whose names start with ‘J’ are grouped together. Another crucial point is that indexes also store pointers to the rows in the table to access other column data.What Exactly Is Stored in a Database Index?You now understand that a database index is created on a table’s column and stores all the values of that column. However, it’s vital to grasp that the database index does not store values from other columns (fields) in the table. For instance, if we create an index on the Employee_Name column, the values of Employee_Age and Employee_Address are not stored in this index. Indeed, if we were to store all other field values in this index as well, it would essentially amount to copying the entire table as an index - occupying excessive space and being highly inefficient.Indexes Store Pointers to Rows in the TableIf we find a record’s value in the index for a column, how do we access the other values of this record? It’s straightforward - the database index also stores pointers to the corresponding rows in the table. A pointer refers to a memory area, recording the reference to the corresponding row’s data on the disk. Thus, in addition to storing column values, the index stores an index to the row data. For example, a value (or node) in the index for the Employee_Name column might be described as (“Jesus”, 0x82829), where 0x82829 is the address of the row containing “Jesus” on the disk. Without this reference, you could only access a single value (“Jesus”), which would be meaningless since you wouldn’t be able to obtain other values for the employee - such as address and age.How Does the Database Know When to Use an Index?When the SQL (SELECT * FROM Employee WHERE Employee_Name = ‘Jesus’) is executed, the database checks whether there is an index on the column involved in the query. Assuming an index exists on the Employee_Name column, the database then determines whether using this index for the query is sensible - as in some scenarios, using an index might be less efficient than a full table scan. For more on these scenarios, consider reading the article: Selectivity in SQLCan You Force the Database to Use an Index?Typically, you don’t tell the database when to use an index - it decides on its own. However, it’s worth noting that in most databases (like Oracle and MYSQL), you can actually specify which index you want to use.How to Create an Index Using SQL:In our previous example, creating an index on the Employee_Name column would involve the following SQL:1CREATE INDEX name_index ON Employee (Employee_Name) How to Create a Composite IndexWe could create a composite index on two columns of the employee table with the following SQL:1CREATE INDEX name_age_index ON Employee (Employee_Name, Employee_Age) What’s a Good Analogy for Database Indexes?A fitting analogy for database indexes is the index of a book. If you have a book about dogs and you wish to find information on ‘Golden Retrievers’, why flip through the entire book when you could refer to the index at the back to see which pages discuss ‘Golden Retrievers’? Similarly, just as a book’s index contains page numbers, a database’s index contains pointers, directing you to the rows containing the values you’re querying in SQL.What Are the Costs of Using Database Indexes?So, what are the downsides to using database indexes? Firstly, indexes occupy space - the larger your table, the more space the index consumes. Secondly, there’s a performance hit (mainly for update operations) when adding, deleting, or updating row data in the table, as the same operations must occur in the index. Remember: creating an index on a column (or columns) requires keeping the column’s data up to date in the index.The general principle is to create an index on a column only if it’s frequently used in queries.","link":"/2023/12/05/DatabaseIndex/"},{"title":"Database View","text":"IntroductionA View in SQL Server can be considered as a virtual table defined on top of it. True to its name, a view offers another entrance to look at data. A conventional view does not store actual data but merely contains a SELECT statement and metadata about the tables involved. Through views, clients no longer need to understand the underlying table structures and their relationships, as views provide a unified interface to access data. Why Use Views? Views abstract the underlying table structures, simplifying data access operations. By hiding the underlying table structures, security is significantly enhanced, allowing users to see only the data provided by the views. Views facilitate permission management by granting users access to views instead of the underlying tables, further strengthening security. Views provide an interface for users to access data. When the underlying tables change, altering the view’s statement to adapt ensures that client programs built on this view remain unaffected. Classification of Views in SQLViews in SQL can be categorized into three types: Regular View Indexed View Partitioned View Let’s discuss these view types in detail. 1) Regular ViewA Regular View is defined by a SELECT statement and includes only its definition and the metadata of the referenced tables, without actually storing data. The template for creating a view as per MSDN is as follows: 1234567891011CREATE VIEW [schema_name.]view_name [(column [, ...n])][WITH <view_attribute> [, ...n]]AS select_statement[WITH CHECK OPTION] [;]<view_attribute> ::={ [ENCRYPTION] [SCHEMABINDING] [VIEW_METADATA]} Parameters are quite straightforward, and here is an explanation for the above: ENCRYPTION: The view is encrypted. Selecting this option prevents modifications. Save the script when creating the view; otherwise, it cannot be modified later. SCHEMABINDING: Binds the view definition to the schema of the referenced tables. If this option is selected, the schema of the referenced tables (such as column data types) cannot be changed casually. To alter the underlying table’s schema, first drop or alter the views bound to it. VIEW_METADATA: An interesting option. If not selected, the metadata returned to the client is that of the tables referenced by the View. If selected, the View’s metadata is returned, making the view appear more like a table to clients. WITH CHECK OPTION: This option is used for restricting data updates through the view, which will be explained further in the section on updating data through views. There are some rules to follow when creating views, aside from complying with the above syntax: In a View, unless the TOP keyword is present, the ORDER BY clause cannot be used (a hack to use ORDER BY is applying TOP 100 percent…). Views must be uniquely named within each Schema. Views nesting cannot exceed 32 levels (in practical work, nesting beyond two levels is discouraged). Keywords like COMPUTE, COMPUTE BY, INTO are not allowed in Views. 2) Indexed ViewAn Indexed View is a regular view with a unique clustered index added to it, effectively turning the view into an object equivalent to a table!Indexed views in SQL Server are conceptually similar to Oracle’s Materialized Views. To understand indexed views, one must first grasp the concept of clustered indexes. Simplified, a clustered index can be thought of as a primary key, with the database’s data physically stored in the table according to the primary key’s order, like a dictionary organized from A to Z. This organization avoids full table scans, thus improving performance. Therefore, a table can have only one clustered index.For indexed views, adding a clustered index means the view no longer just contains a SELECT statement and table metadata; the indexed view physically stores data in the database and keeps it synchronized with the underlying tables.Understanding the principle behind indexed views reveals their significant performance boost for LDAP, involving massive data analysis and queries, especially when indexed views contain aggregate functions and involve costly JOINs. Since the results of aggregate functions are physically stored in the indexed view, it’s not necessary to perform aggregation every time the view is used, markedly improving performance.However, every time an underlying table involved in an indexed view is updated, inserted, or deleted, SQL Server needs to identify the changed rows to synchronize the indexed view. Thus, for OLTP systems with frequent modifications, the performance might degrade due to the extensive synchronization required.3) Partitioned ViewPartitioned views, from a micro-implementation perspective, return data sets obtained by UNION-joining several parallel tables (i.e., tables with the same structure - columns and data types, but storing different row sets).There are generally two types of partitioned views: Local Partitioned View Distributed Partitioned View Since local partitioned views are mainly for backward compatibility with versions before SQL Server 2005, this article will focus on distributed partitioned views. Distributed partitioned views essentially connect parallel data sets obtained from different or the same data sources. The biggest advantage of using distributed partitioned views is performance enhancement. For example, in the scenario above, if I only want to retrieve information for EmployeeID=8 using a distributed view, SQL Server can intelligently scan only Table 2 that contains EmployeeID=8, avoiding a full table scan. This significantly reduces IO operations, thereby enhancing performance.It’s important to note that the primary keys involved in the tables of a distributed partitioned view cannot overlap. For example, if Table A’s ContactID ranges from 1-4, then Table B’s ContactID cannot range from 2-8.Another point to note is that Check constraints should be added to the primary keys of the distributed partitioned view, enabling the SQL Server’s query analyzer to know which table to scan. Here’s an example:In the Microsoft sample database AdventureWorks, I store data from the first 100 rows and rows 100-200 into two tables, Employee100 and Employee200, respectively:1234567891011-- Create Employee100SELECT TOP 100 * INTO Employee100FROM HumanResources.Employee ORDER BY EmployeeID-- Create Employee200SELECT * INTO Employee200FROM (SELECT TOP 100 *FROM HumanResources.Employee WHERE EmployeeID NOT IN (SELECT TOP 100 EmployeeID FROM HumanResources.Employee ORDER BY EmployeeID)ORDER BY HumanResources.Employee.EmployeeID) AS e Now, let’s create a distributed partitioned view:12345CREATE VIEW v_part_view_testASSELECT * FROM Employee100UNION SELECT * FROM Employee200 When querying this view:12SELECT * FROM v_part_view_testWHERE EmployeeID=105 So, when different data tables are placed on different servers or use a RAID5 disk array, distributed partitioned views can further enhance query performance.Can using distributed partitioned views always improve performance? Absolutely not. If the query involves aggregate functions, especially with DISTINCT in the aggregate functions, or sorts without a WHERE condition, it’s definitely a performance killer. Aggregate functions require scanning all tables in the distributed partitioned view, then performing UNION operations before calculating, which can significantly impact performance.Updating Data Through ViewsI do not recommend updating data through views since views cannot accept parameters. I prefer using stored procedures for implementation.Updating data through views is done in the same manner as updating data in tables (as mentioned before, a view can be considered as a virtual table, and if it’s an indexed view, it’s practically a table).When updating data through views, keep in mind: At least one user table must follow the FROM clause in the view. Regardless of how many tables the view’s query involves, only one table’s data can be updated at a time. Columns calculated through expressions, constant columns, and columns resulting from aggregate functions cannot be updated. Columns not affected by Group By, Having, and Distinct keywords cannot be updated. Tips in ViewsTo find a view’s definition through its name:12SELECT * FROM sys.sql_modulesWHERE object_id=OBJECT_ID('view_name') As mentioned, regular views only store the SELECT statement and referenced tables’ metadata. When the underlying table data changes, sometimes the view’s table metadata is not synchronized in time. Use the following code for manual synchronization:1EXEC sp_refreshview 'view_name' Best Practices in ViewsThese are some of my personal experiences, and I welcome additions: Ensure the SELECT statement in the View is optimized for performance (seems obvious, but truths are often simple). Avoid nesting views, if necessary, at most nest one level. Use stored procedures and user-defined functions instead of Views when possible, as stored procedures cache execution plans, offer better performance, and have fewer restrictions. Do not use aggregate functions in partitioned views, especially when they include DISTINCT. If possible, include WHERE clauses inside the view rather than outside (because calling a view returns all rows, then filters, which is a performance killer, especially if you also add an ORDER BY…). ConclusionThis article detailed the three types of views, each with its own use cases. Properly used, they can enhance performance, but improper use can drag down performance.Remember the adage: “Practice makes perfect”… With practice and thought, there will surely be gains.","link":"/2023/12/17/DatabseView/"},{"title":"Why Does MySQL Use B+ Trees Instead of Skip Lists for Indexing?","text":"=============================================================== Today I saw a very useful and clear video about the explanation of B+ tree and skip lists in Database, so I want to write a blog to record my understanding. When we think about MySQL tables, they seem to be simply storing rows of data, similar to a spreadsheet. Directly traversing these rows of data gives us a performance of O(n), which is quite slow. To accelerate queries, B+ trees are used for indexing, optimizing the query performance to O(log n). But this raises a question: there are many data structures with log(n) level query performance, such as skip lists used in Redis’s zset, which are also log(n) and even simpler to implement. So why doesn’t MySQL use skip lists for indexing? The Structure of B+ TreesGenerally, B+ trees are a multi-level structure composed of multiple pages, each being 16Kb. For primary key indexes, the leaf nodes at the lowest level hold row data, while the internal nodes hold index information (primary key id and page number) to speed up queries. For instance, if we want to find row data with id=5, we start from the top-level page records. Each record contains a primary key id and a page number (page address). Follow the yellow arrow; to the left, the smallest id is 1, to the right, the smallest id is 7. Therefore, if the data with id=5 exists, it must be on the left side. Following the record’s page address, we arrive at page 6, then determine id=5>4, so it must be on the right side in page 105. In page 105, although there are multiple rows of data, they are not traversed one by one. The page also has a page directory, which can speed up the query for row data through binary search, hence finding the row data with id=5 and completing the query. As we can see, B+ trees use a space-for-time approach (creating a batch of internal nodes to store index information), optimizing the query time complexity from O(n) to O(log n). The Structure of Skip ListsHaving looked at B+ trees, let’s examine skip lists. Similarly, they are used to store rows of data. We can string them together in a linked list. To query a node in the list, the time complexity is O(n), which is too high. Thus, some linked list nodes are elevated to construct a new list. When I want to query data, I first check the upper list, which quickly tells me the range where the data falls, and then I jump to the next level for querying. This significantly narrows down the search scope. For example, to query data with id=10, we first traverse the upper layer, sequentially assessing 1,6,12, and quickly determine that 10 is between 6 and 12. Then, jumping down a level, we can confirm the position of id=10 after traversing 6,7,8,9,10. The search range is effectively halved. If just two layers of linked lists can halve the search range, adding more layers would be even better, right? Thus, the skip list becomes multi-layered. To continue with our example of querying data with id=10, we would only need to search through 1,6,9,10 to find it, which is quicker than with two layers. It’s evident that skip lists also trade space for time to enhance query performance. The time complexity is log(n). The Differences Between B+ Trees and Skip ListsAs we can see, the lowest level of both B+ trees and skip lists contains all the data, which is sequential and suitable for range queries. The upper levels are constructed to improve search performance. These two structures are strikingly similar. However, they differ in how they handle data insertion and deletion. Let’s discuss this with data insertion as an example. B+ Tree Data InsertionA B+ tree is essentially a multi-way balanced binary tree. The key lies in the word “balanced,” meaning for a multi-way tree structure, the heights of the subtrees are as consistent as possible (generally differing by no more than one level). This ensures that no matter which subtree branch you search, the number of searches won’t vary much. As new data is continuously inserted into the database table, the B+ tree adjusts and splits data pages to maintain balance. We know that a B+ tree is divided into leaf nodes and internal nodes. When a piece of data is inserted, both the leaf node and its upper index node (internal node), which have a maximum capacity of 16k, may become full. For simplicity, let’s assume a data page can only hold three rows of data or indexes. There are three scenarios when inserting data, depending on whether the data page is full: Neither the leaf node nor the index node is full. This is the simplest case; just insert directly into the leaf node. The leaf node is full, but the index node is not. In this case, the leaf node must be split, and the index node must add new index information. Both the leaf node and the index node are full. Both leaf and index nodes need to be split, and a new layer of indexing must be added above. As we can see, only when both the leaf and index nodes are full does the B+ tree consider adding a new layer of nodes. From previous articles, we know that it takes roughly 2 million entries to fill a three-layer B+ tree. Skip List Data InsertionA skip list is also multi-layered, and when new data is inserted, the lowest layer of the list needs to add the data. At this point, is it necessary to add data for indexing in the upper layers? This purely depends on a random function. Theoretically, to achieve the effect of binary search, each layer’s node count should be half of the layer below it. That is to say, now that new data has been inserted, there’s a 50% chance it will need to be indexed in the second layer, a 25% chance it will need to be indexed in the third layer, and so on up to the top layer. For example, if data with id=6 is inserted and the random function returns the third layer (a 25% chance), then data must be inserted from the lowest to the third layer in the skip list. If this random function is designed as described, when there’s a sufficiently large sample size of data, the distribution will match our ideal “binary” distribution. Unlike the B+ tree, whether a skip list adds layers depends purely on the random function and has nothing to do with neighboring nodes. Why Does MySQL Use B+ Trees Instead of Skip Lists?B+ trees are multi-way tree structures. Each node is a 16k data page that can hold a lot of index information, so the fan-out is very high. About three layers can store approximately 2 million entries (knowing the conclusion is enough, if you want to know the reason, you can refer to previous articles). That is, to query data, if those data pages are all on disk, then at most three disk IOs are needed. A skip list is a linked list structure, with one node per piece of data. If the bottom layer needs to store 2 million entries and achieve binary search efficiency with each query, 2 million is roughly in the ballpark of 2 to the power of 24, meaning the skip list would be about 24 layers high. In the worst case, these 24 layers of data would be scattered across different data pages, or in other words, searching for data would entail 24 disk IOs. Therefore, to store the same amount of data, the height of the B+ tree is less than that of the skip list. If placed in a MySQL database, this means fewer disk IOs and thus faster B+ tree queries. As for write operations, B+ trees need to split and merge index data pages, whereas skip lists independently insert data and determine the number of layers based on a random function, without the cost of rotations and maintaining balance. Therefore, the write performance of skip lists is better than that of B+ trees. In fact, MySQL’s storage engine can be changed. It used to be MyISAM, and then InnoDB came along, both using B+ trees for indexing. This means you could theoretically build a storage engine with skip list indexing for MySQL. In fact, Facebook created a RocksDB storage engine that uses skip lists. To put it simply, its write performance is indeed better than InnoDB’s, but its read performance is quite a bit worse. Why Does Redis Use Skip Lists Instead of B+ Trees or Binary Trees?Redis supports a variety of data structures, including a sorted set, also known as ZSET, which is internally implemented with skip lists. So why use skip lists instead of B+ trees or other structures? This question almost always comes up in interviews. Although I’m very familiar with it, every time I have to pretend I’ve never thought about it before and come up with the answer on the spot. Really, it’s a test of acting skills. As we know, Redis is a pure in-memory database. Read and write operations are performed in memory and have nothing to do with the disk, so the height of the structure is no longer a disadvantage for skip lists. Moreover, as mentioned earlier, B+ trees involve a series of merge and split operations. If we were to use a red-black tree or other AVL tree, there would be various rotations, all aimed at maintaining the balance of the tree. However, when inserting data into a skip list, a simple random decision is made as to whether to add an index above, without any consideration for neighboring nodes and thus avoiding the cost of rotations and balancing. Therefore, Redis chose skip lists over B+ trees. ConclusionB+ trees are multi-way balanced search trees with a high fan-out, requiring only about three layers to store approximately 2 million entries. Under similar conditions, skip lists would need about 24 layers. Assuming layer height corresponds to disk IO, B+ trees would have better read performance than skip lists, which is why MySQL chose B+ trees for indexing. Redis operations are conducted entirely in memory, not involving disk IO. Skip lists are simple to implement, and compared to B+ trees, AVL trees, and others, they avoid the overhead of rotating tree structures, which is why Redis uses skip lists to implement ZSET instead of tree structures. The storage engine RocksDB uses skip lists internally. Compared to InnoDB, which uses B+ trees, although it has better write performance, its read performance is indeed worse. In scenarios where reads are more frequent than writes, B+ trees are still considered superior.","link":"/2024/02/21/databaseB+tree/"},{"title":"RESTful API","text":"Understanding RESTful in Web DevelopmentWeb development has evolved significantly with the advent of Web 2.0, enabling more dynamic interactions between users and websites. This evolution has led to a shift in development methodologies, particularly with the adoption of the front-end and back-end separation model. In this blog, we will delve into the concept of RESTful architecture, its relevance in the current web development landscape, and how it facilitates the development of both web and mobile applications. The Current State of Web DevelopmentTo fully comprehend RESTful architecture, it’s essential to understand the current state of web development. The separation of the front-end and back-end is a fundamental paradigm in modern web development. This approach involves creating two distinct servers: Front-end Server: Handles the user interface and interaction. Back-end Server: Manages the database and server-side logic. This separation offers several advantages, particularly in the era of Web 2.0 development. It accommodates the development needs of not just PC platforms but also supports mobile app development. By separating concerns, mobile developers can directly interact with back-end APIs, sending requests and processing JSON responses from the back-end server. This efficiency enables more streamlined and flexible development across different platforms. Understanding RESTful ArchitectureThe application of RESTful principles is based on standardizing back-end server interfaces. Before the widespread adoption of RESTful APIs, developers often created APIs in a somewhat ad-hoc fashion. For a simple function like ‘add,’ there might be several, if not dozens, of different APIs, each developed according to the individual developer’s preferences. This lack of standardization led to complexity and inefficiency, hindering collaboration and scaling. RESTful architecture aims to resolve these issues by introducing a set of guidelines for API design. REST, or Representational State Transfer, is an architectural style that uses HTTP requests to communicate between clients and servers. It relies on stateless, client-server, cacheable communications protocol – the HTTP protocol. RESTful APIs are designed around resources, which are any kind of object, data, or service that can be accessed by the client. A resource has a URI (Uniform Resource Identifier) and is manipulated using standard HTTP methods (GET, POST, PUT, DELETE, etc.). Key Principles of RESTful ArchitectureRESTful architecture is built upon several key principles: Uniform Interface: This principle ensures a standardized way of interacting with a given resource through a specific URI. This uniformity simplifies the architecture, making the interactions between the client and server more predictable and streamlined. Stateless Operations: Each request from client to server must contain all the information needed to understand and complete the request. The server does not store any session information about the client. Cacheable Responses: To improve network efficiency, responses from the server should be designed to be cacheable. This reduces the load on the server and improves the overall responsiveness of the application. Client-Server Decoupling: The client and server evolve independently, with the client not concerned about data storage, which remains internal to the server, and the server not concerned about the user interface or user state, which remains internal to the client. Layered System: The client cannot ordinarily tell whether it is connected directly to the end server or to an intermediary along the way. Intermediary servers can improve system scalability by enabling load-balancing and by providing shared caches. Code on Demand (optional): Servers can temporarily extend or customize the functionality of a client by transferring executable code. ConclusionRESTful architecture provides a robust and flexible framework for web and mobile application development. By adhering to its principles, developers can create scalable, efficient, and easy-to-maintain APIs. The separation of front-end and back-end, coupled with the adoption of RESTful principles, signifies a mature approach to developing applications that cater to the needs of the modern web and mobile landscape.","link":"/2024/01/20/restfulAPI/"},{"title":"Merge Intervals","text":"A coding challenge about Intervals MergingI finished an interesting coding problem and I want to share it here. Interval merging refers to combining several overlapping intervals into a single interval. This process is useful in various applications, such as consolidating time slots or ranges of numbers. Python Function to Merge IntervalsBelow is a Python function that merges overlapping intervals: 123456789101112131415161718def merge_intervals(intervals): # Sort the intervals by their starting point intervals = sorted(intervals, key=lambda x: x[0]) merged = [] for interval in intervals: # If the current interval overlaps with the last interval in merged, merge them if merged and interval[0] <= merged[-1][1]: merged[-1][1] = max(merged[-1][1], interval[1]) else: # If they don't overlap, add the current interval to merged merged.append(interval) return merged # Example usageintervals = [[1, 3], [2, 6], [8, 10], [15, 18]]merged_intervals = merge_intervals(intervals)print(merged_intervals)# Output: [[1, 6], [8, 10], [15, 18]] ProcessThe process of merging intervals involves: First, sort the list of intervals based on the start point of each interval. Then, iterate through the sorted list of intervals. For each interval: If it overlaps with the previous interval in the merged list, combine them into a single interval by updating the end point of the last interval in the merged list to the maximum end point of both intervals. If there is no overlap, simply add the current interval to the merged list. Return the merged list of intervals as the result. By following these steps, you can efficiently merge overlapping intervals into a consolidated list.","link":"/2024/03/17/MergeInterval/"}],"tags":[{"name":"System Design","slug":"System-Design","link":"/tags/System-Design/"},{"name":"Database","slug":"Database","link":"/tags/Database/"},{"name":"Algorithms","slug":"Algorithms","link":"/tags/Algorithms/"},{"name":"RESTful API","slug":"RESTful-API","link":"/tags/RESTful-API/"},{"name":"offer","slug":"offer","link":"/tags/offer/"}],"categories":[{"name":"Life","slug":"Life","link":"/categories/Life/"},{"name":"Technical skills","slug":"Technical-skills","link":"/categories/Technical-skills/"}]}