How LINQ to Object statements work
- by rajbk
This post goes into detail as to now LINQ statements work when querying a collection of objects. This topic assumes you have an understanding of how generics, delegates, implicitly typed variables, lambda expressions, object/collection initializers, extension methods and the yield statement work. I would also recommend you read my previous two posts: Using Delegates in C# Part 1 Using Delegates in C# Part 2 We will start by writing some methods to filter a collection of data. Assume we have an Employee class like so: 1: public class Employee {
2: public int ID { get; set;}
3: public string FirstName { get; set;}
4: public string LastName {get; set;}
5: public string Country { get; set; }
6: }
and a collection of employees like so:
1: var employees = new List<Employee> {
2: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
3: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
4: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
5: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" },
6: };
Filtering
We wish to find all employees that have an even ID. We could start off by writing a method that takes in a list of employees and returns a filtered list of employees with an even ID.
1: static List<Employee> GetEmployeesWithEvenID(List<Employee> employees) {
2: var filteredEmployees = new List<Employee>();
3: foreach (Employee emp in employees) {
4: if (emp.ID % 2 == 0) {
5: filteredEmployees.Add(emp);
6: }
7: }
8: return filteredEmployees;
9: }
The method can be rewritten to return an IEnumerable<Employee> using the yield return keyword.
1: static IEnumerable<Employee> GetEmployeesWithEvenID(IEnumerable<Employee> employees) {
2: foreach (Employee emp in employees) {
3: if (emp.ID % 2 == 0) {
4: yield return emp;
5: }
6: }
7: }
We put these together in a console application.
1: using System;
2: using System.Collections.Generic;
3: //No System.Linq
4:
5: public class Program
6: {
7: [STAThread]
8: static void Main(string[] args)
9: {
10: var employees = new List<Employee> {
11: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
12: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
13: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
14: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" },
15: };
16: var filteredEmployees = GetEmployeesWithEvenID(employees);
17:
18: foreach (Employee emp in filteredEmployees) {
19: Console.WriteLine("ID {0} First_Name {1} Last_Name {2} Country {3}",
20: emp.ID, emp.FirstName, emp.LastName, emp.Country);
21: }
22:
23: Console.ReadLine();
24: }
25:
26: static IEnumerable<Employee> GetEmployeesWithEvenID(IEnumerable<Employee> employees) {
27: foreach (Employee emp in employees) {
28: if (emp.ID % 2 == 0) {
29: yield return emp;
30: }
31: }
32: }
33: }
34:
35: public class Employee {
36: public int ID { get; set;}
37: public string FirstName { get; set;}
38: public string LastName {get; set;}
39: public string Country { get; set; }
40: }
Output:
ID 2 First_Name Jim Last_Name Ashlock Country UK
ID 4 First_Name Jill Last_Name Anderson Country AUS
Our filtering method is too specific. Let us change it so that it is capable of doing different types of filtering and lets give our method the name Where ;-)
We will add another parameter to our Where method. This additional parameter will be a delegate with the following declaration.
public delegate bool Filter(Employee emp);
The idea is that the delegate parameter in our Where method will point to a method that contains the logic to do our filtering thereby freeing our Where method from any dependency. The method is shown below:
1: static IEnumerable<Employee> Where(IEnumerable<Employee> employees, Filter filter) {
2: foreach (Employee emp in employees) {
3: if (filter(emp)) {
4: yield return emp;
5: }
6: }
7: }
Making the change to our app, we create a new instance of the Filter delegate on line 14 with a target set to the method EmployeeHasEvenId. Running the code will produce the same output.
1: public delegate bool Filter(Employee emp);
2:
3: public class Program
4: {
5: [STAThread]
6: static void Main(string[] args)
7: {
8: var employees = new List<Employee> {
9: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
10: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
11: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
12: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
13: };
14: var filterDelegate = new Filter(EmployeeHasEvenId);
15: var filteredEmployees = Where(employees, filterDelegate);
16:
17: foreach (Employee emp in filteredEmployees) {
18: Console.WriteLine("ID {0} First_Name {1} Last_Name {2} Country {3}",
19: emp.ID, emp.FirstName, emp.LastName, emp.Country);
20: }
21: Console.ReadLine();
22: }
23:
24: static bool EmployeeHasEvenId(Employee emp) {
25: return emp.ID % 2 == 0;
26: }
27:
28: static IEnumerable<Employee> Where(IEnumerable<Employee> employees, Filter filter) {
29: foreach (Employee emp in employees) {
30: if (filter(emp)) {
31: yield return emp;
32: }
33: }
34: }
35: }
36:
37: public class Employee {
38: public int ID { get; set;}
39: public string FirstName { get; set;}
40: public string LastName {get; set;}
41: public string Country { get; set; }
42: }
Lets use lambda expressions to inline the contents of the EmployeeHasEvenId method in place of the method. The next code snippet shows this change (see line 15). For brevity, the Employee class declaration has been skipped.
1: public delegate bool Filter(Employee emp);
2:
3: public class Program
4: {
5: [STAThread]
6: static void Main(string[] args)
7: {
8: var employees = new List<Employee> {
9: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
10: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
11: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
12: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
13: };
14: var filterDelegate = new Filter(EmployeeHasEvenId);
15: var filteredEmployees = Where(employees, emp => emp.ID % 2 == 0);
16:
17: foreach (Employee emp in filteredEmployees) {
18: Console.WriteLine("ID {0} First_Name {1} Last_Name {2} Country {3}",
19: emp.ID, emp.FirstName, emp.LastName, emp.Country);
20: }
21: Console.ReadLine();
22: }
23:
24: static bool EmployeeHasEvenId(Employee emp) {
25: return emp.ID % 2 == 0;
26: }
27:
28: static IEnumerable<Employee> Where(IEnumerable<Employee> employees, Filter filter) {
29: foreach (Employee emp in employees) {
30: if (filter(emp)) {
31: yield return emp;
32: }
33: }
34: }
35: }
36:
The output displays the same two employees.
Our Where method is too restricted since it works with a collection of Employees only. Lets change it so that it works with any IEnumerable<T>. In addition, you may recall from my previous post, that .NET 3.5 comes with a lot of predefined delegates including
public delegate TResult Func<T, TResult>(T arg);
We will get rid of our Filter delegate and use the one above instead. We apply these two changes to our code.
1: public class Program
2: {
3: [STAThread]
4: static void Main(string[] args)
5: {
6: var employees = new List<Employee> {
7: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
8: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
9: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
10: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
11: };
12:
13: var filteredEmployees = Where(employees, emp => emp.ID % 2 == 0);
14:
15: foreach (Employee emp in filteredEmployees) {
16: Console.WriteLine("ID {0} First_Name {1} Last_Name {2} Country {3}",
17: emp.ID, emp.FirstName, emp.LastName, emp.Country);
18: }
19: Console.ReadLine();
20: }
21:
22: static IEnumerable<T> Where<T>(IEnumerable<T> source, Func<T, bool> filter) {
23: foreach (var x in source) {
24: if (filter(x)) {
25: yield return x;
26: }
27: }
28: }
29: }
We have successfully implemented a way to filter any IEnumerable<T> based on a filter criteria.
Projection
Now lets enumerate on the items in the IEnumerable<Employee> we got from the Where method and copy them into a new IEnumerable<EmployeeFormatted>. The EmployeeFormatted class will only have a FullName and ID property.
1: public class EmployeeFormatted {
2: public int ID { get; set; }
3: public string FullName {get; set;}
4: }
We could “project” our existing IEnumerable<Employee> into a new collection of IEnumerable<EmployeeFormatted> with the help of a new method. We will call this method Select ;-)
1: static IEnumerable<EmployeeFormatted> Select(IEnumerable<Employee> employees) {
2: foreach (var emp in employees) {
3: yield return new EmployeeFormatted {
4: ID = emp.ID,
5: FullName = emp.LastName + ", " + emp.FirstName
6: };
7: }
8: }
The changes are applied to our app.
1: public class Program
2: {
3: [STAThread]
4: static void Main(string[] args)
5: {
6: var employees = new List<Employee> {
7: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
8: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
9: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
10: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
11: };
12:
13: var filteredEmployees = Where(employees, emp => emp.ID % 2 == 0);
14: var formattedEmployees = Select(filteredEmployees);
15:
16: foreach (EmployeeFormatted emp in formattedEmployees) {
17: Console.WriteLine("ID {0} Full_Name {1}",
18: emp.ID, emp.FullName);
19: }
20: Console.ReadLine();
21: }
22:
23: static IEnumerable<T> Where<T>(IEnumerable<T> source, Func<T, bool> filter) {
24: foreach (var x in source) {
25: if (filter(x)) {
26: yield return x;
27: }
28: }
29: }
30:
31: static IEnumerable<EmployeeFormatted> Select(IEnumerable<Employee> employees) {
32: foreach (var emp in employees) {
33: yield return new EmployeeFormatted {
34: ID = emp.ID,
35: FullName = emp.LastName + ", " + emp.FirstName
36: };
37: }
38: }
39: }
40:
41: public class Employee {
42: public int ID { get; set;}
43: public string FirstName { get; set;}
44: public string LastName {get; set;}
45: public string Country { get; set; }
46: }
47:
48: public class EmployeeFormatted {
49: public int ID { get; set; }
50: public string FullName {get; set;}
51: }
Output:
ID 2 Full_Name Ashlock, Jim
ID 4 Full_Name Anderson, Jill
We have successfully selected employees who have an even ID and then shaped our data with the help of the Select method so that the final result is an IEnumerable<EmployeeFormatted>.
Lets make our Select method more generic so that the user is given the freedom to shape what the output would look like. We can do this, like before, with lambda expressions. Our Select method is changed to accept a delegate as shown below. TSource will be the type of data that comes in and TResult will be the type the user chooses (shape of data) as returned from the selector delegate.
1:
2: static IEnumerable<TResult> Select<TSource, TResult>(IEnumerable<TSource> source, Func<TSource, TResult> selector) {
3: foreach (var x in source) {
4: yield return selector(x);
5: }
6: }
We see the new changes to our app. On line 15, we use lambda expression to specify the shape of the data. In this case the shape will be of type EmployeeFormatted.
1:
2: public class Program
3: {
4: [STAThread]
5: static void Main(string[] args)
6: {
7: var employees = new List<Employee> {
8: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
9: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
10: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
11: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
12: };
13:
14: var filteredEmployees = Where(employees, emp => emp.ID % 2 == 0);
15: var formattedEmployees = Select(filteredEmployees, (emp) =>
16: new EmployeeFormatted {
17: ID = emp.ID,
18: FullName = emp.LastName + ", " + emp.FirstName
19: });
20:
21: foreach (EmployeeFormatted emp in formattedEmployees) {
22: Console.WriteLine("ID {0} Full_Name {1}",
23: emp.ID, emp.FullName);
24: }
25: Console.ReadLine();
26: }
27:
28: static IEnumerable<T> Where<T>(IEnumerable<T> source, Func<T, bool> filter) {
29: foreach (var x in source) {
30: if (filter(x)) {
31: yield return x;
32: }
33: }
34: }
35:
36: static IEnumerable<TResult> Select<TSource, TResult>(IEnumerable<TSource> source, Func<TSource, TResult> selector) {
37: foreach (var x in source) {
38: yield return selector(x);
39: }
40: }
41: }
The code outputs the same result as before. On line 14 we filter our data and on line 15 we project our data.
What if we wanted to be more expressive and concise? We could combine both line 14 and 15 into one line as shown below. Assuming you had to perform several operations like this on our collection, you would end up with some very unreadable code!
1: var formattedEmployees = Select(Where(employees, emp => emp.ID % 2 == 0), (emp) =>
2: new EmployeeFormatted {
3: ID = emp.ID,
4: FullName = emp.LastName + ", " + emp.FirstName
5: });
A cleaner way to write this would be to give the appearance that the Select and Where methods were part of the IEnumerable<T>. This is exactly what extension methods give us. Extension methods have to be defined in a static class. Let us make the Select and Where extension methods on IEnumerable<T>
1: public static class MyExtensionMethods {
2: static IEnumerable<T> Where<T>(this IEnumerable<T> source, Func<T, bool> filter) {
3: foreach (var x in source) {
4: if (filter(x)) {
5: yield return x;
6: }
7: }
8: }
9:
10: static IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector) {
11: foreach (var x in source) {
12: yield return selector(x);
13: }
14: }
15: }
The creation of the extension method makes the syntax much cleaner as shown below. We can write as many extension methods as we want and keep on chaining them using this technique.
1: var formattedEmployees = employees
2: .Where(emp => emp.ID % 2 == 0)
3: .Select (emp => new EmployeeFormatted { ID = emp.ID, FullName = emp.LastName + ", " + emp.FirstName });
Making these changes and running our code produces the same result.
1: using System;
2: using System.Collections.Generic;
3:
4: public class Program
5: {
6: [STAThread]
7: static void Main(string[] args)
8: {
9: var employees = new List<Employee> {
10: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
11: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
12: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
13: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
14: };
15:
16: var formattedEmployees = employees
17: .Where(emp => emp.ID % 2 == 0)
18: .Select (emp =>
19: new EmployeeFormatted {
20: ID = emp.ID,
21: FullName = emp.LastName + ", " + emp.FirstName
22: }
23: );
24:
25: foreach (EmployeeFormatted emp in formattedEmployees) {
26: Console.WriteLine("ID {0} Full_Name {1}",
27: emp.ID, emp.FullName);
28: }
29: Console.ReadLine();
30: }
31: }
32:
33: public static class MyExtensionMethods {
34: static IEnumerable<T> Where<T>(this IEnumerable<T> source, Func<T, bool> filter) {
35: foreach (var x in source) {
36: if (filter(x)) {
37: yield return x;
38: }
39: }
40: }
41:
42: static IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector) {
43: foreach (var x in source) {
44: yield return selector(x);
45: }
46: }
47: }
48:
49: public class Employee {
50: public int ID { get; set;}
51: public string FirstName { get; set;}
52: public string LastName {get; set;}
53: public string Country { get; set; }
54: }
55:
56: public class EmployeeFormatted {
57: public int ID { get; set; }
58: public string FullName {get; set;}
59: }
Let’s change our code to return a collection of anonymous types and get rid of the EmployeeFormatted type. We see that the code produces the same output.
1: using System;
2: using System.Collections.Generic;
3:
4: public class Program
5: {
6: [STAThread]
7: static void Main(string[] args)
8: {
9: var employees = new List<Employee> {
10: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
11: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
12: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
13: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
14: };
15:
16: var formattedEmployees = employees
17: .Where(emp => emp.ID % 2 == 0)
18: .Select (emp =>
19: new {
20: ID = emp.ID,
21: FullName = emp.LastName + ", " + emp.FirstName
22: }
23: );
24:
25: foreach (var emp in formattedEmployees) {
26: Console.WriteLine("ID {0} Full_Name {1}",
27: emp.ID, emp.FullName);
28: }
29: Console.ReadLine();
30: }
31: }
32:
33: public static class MyExtensionMethods {
34: public static IEnumerable<T> Where<T>(this IEnumerable<T> source, Func<T, bool> filter) {
35: foreach (var x in source) {
36: if (filter(x)) {
37: yield return x;
38: }
39: }
40: }
41:
42: public static IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector) {
43: foreach (var x in source) {
44: yield return selector(x);
45: }
46: }
47: }
48:
49: public class Employee {
50: public int ID { get; set;}
51: public string FirstName { get; set;}
52: public string LastName {get; set;}
53: public string Country { get; set; }
54: }
To be more expressive, C# allows us to write our extension method calls as a query expression. Line 16 can be rewritten a query expression like so:
1: var formattedEmployees = from emp in employees
2: where emp.ID % 2 == 0
3: select new {
4: ID = emp.ID,
5: FullName = emp.LastName + ", " + emp.FirstName
6: };
When the compiler encounters an expression like the above, it simply rewrites it as calls to our extension methods.
So far we have been using our extension methods. The System.Linq namespace contains several extension methods for objects that implement the IEnumerable<T>. You can see a listing of these methods in the Enumerable class in the System.Linq namespace.
Let’s get rid of our extension methods (which I purposefully wrote to be of the same signature as the ones in the Enumerable class) and use the ones provided in the Enumerable class. Our final code is shown below:
1: using System;
2: using System.Collections.Generic;
3: using System.Linq; //Added
4:
5: public class Program
6: {
7: [STAThread]
8: static void Main(string[] args)
9: {
10: var employees = new List<Employee> {
11: new Employee { ID = 1, FirstName = "John", LastName = "Wright", Country = "USA" },
12: new Employee { ID = 2, FirstName = "Jim", LastName = "Ashlock", Country = "UK" },
13: new Employee { ID = 3, FirstName = "Jane", LastName = "Jackson", Country = "CHE" },
14: new Employee { ID = 4, FirstName = "Jill", LastName = "Anderson", Country = "AUS" }
15: };
16:
17: var formattedEmployees = from emp in employees
18: where emp.ID % 2 == 0
19: select new {
20: ID = emp.ID,
21: FullName = emp.LastName + ", " + emp.FirstName
22: };
23:
24: foreach (var emp in formattedEmployees) {
25: Console.WriteLine("ID {0} Full_Name {1}",
26: emp.ID, emp.FullName);
27: }
28: Console.ReadLine();
29: }
30: }
31:
32: public class Employee {
33: public int ID { get; set;}
34: public string FirstName { get; set;}
35: public string LastName {get; set;}
36: public string Country { get; set; }
37: }
38:
39: public class EmployeeFormatted {
40: public int ID { get; set; }
41: public string FullName {get; set;}
42: }
This post has shown you a basic overview of LINQ to Objects work by showning you how an expression is converted to a sequence of calls to extension methods when working directly with objects. It gets more interesting when working with LINQ to SQL where an expression tree is constructed – an in memory data representation of the expression. The C# compiler compiles these expressions into code that builds an expression tree at runtime. The provider can then traverse the expression tree and generate the appropriate SQL query. You can read more about expression trees in this MSDN article.
